Brian Edminston
Every piece of advice out there about alert fatigue says the same thing. Reduce the volume. Suppress the noisy rule. Turn something off until the queue gets shorter. I run detection engineering here at OneAxiom, and I think that instinct is exactly backwards.
My short answer is no, don’t focus only on cutting the volume. The real fix is working on alert quality, in other words, understanding which alerts are actually worth investigating.
But let’s keep digging.
What alert fatigue actually costs
First, the plain version of the problem.
Alert fatigue is what happens when you see the same kind of alert over and over, day after day, and you eventually stop investigating it the way you’re supposed to. You start closing things out instead of reading the details, because you already know what it’s going to say.
That’s the moment a visibility gap opens. Something that looked routine on day forty turns out not to be routine at all, and nobody catches it until it’s already a problem.
The scale of this isn’t a guess:
- SANS’ most recent SOC survey found that two thirds of security teams can’t keep pace with the alert volume hitting them, and most say they’re overwhelmed by data volume on top of that.
- Researchers at the University of Illinois studied years of real SOC logs instead of survey responses and found analysts facing tens of thousands of alerts a day, with only a sliver of a percent tied to an actual attack.
- Ponemon’s research puts a number on what that does to people: roughly a quarter of an analyst’s working hour goes to chasing something that turns out to be nothing, and a separate study found most analysts have genuinely considered quitting over the burnout it causes.
Security inherited a humane problem: more monitoring, more noise, the same number of humans trying to keep up with it.
Why turning off a noisy alert won’t solve the root cause
Where I part ways with almost everyone writing about this is what you do next. The common answer is: cut the volume. Mine is: you don’t get to turn something off just because it’s noisy, not if it’s still looking at activity that genuinely needs a human eye on it.
What you do instead is adjust the rule, narrower or broader, depending on what it’s missing or over-catching, so the thing it’s watching for gets sharper. The goal is to get sharper, not quieter.
There’s a real technical difference between tuning a detection and suppressing its output, and it’s not a small one.
Tuning changes what the rule actually catches. Suppression just hides what it already caught. Elastic draws that line explicitly in their own documentation, and Microsoft tells its customers to use suppression with real caution, reserved for known, specific cases, not as a general way to make a dashboard look calmer.
If your fix for alert fatigue is quietly filtering more things out after they’ve already fired, you haven’t solved the problem. You’ve just stopped looking at it.What high-fidelity detection actually means
I don’t believe we need a thousand alerts if 999 of them are reporting nothing. What I want is what we call high-fidelity detection: if it fires, it gets investigated, no exceptions.
That sometimes means more alerts, not fewer, because a properly narrowed rule catches more of what it’s actually supposed to catch. A quiet queue was never the goal, but a queue where everything in it deserves to be there is.
Here’s what that looks like at actual scale.
Last time I counted, we had just over 850 live rules running in our SIEM with a few hundred more in development. Every one of those rules earns its place by catching something specific, and the ones that stop earning it get rewritten or retired, not left running for the sake of coverage on paper.
A big rule count that isn’t maintained this way is just noise with a higher headcount. A small one can hide the same problem. The number only means anything next to the discipline of pruning it.
You don’t have to trust my word for it. A study out of Oxford, presented at USENIX Security in 2022, interviewed dozens of SOC analysts across seven different centers and found something that cuts against the usual complaint about false positives.
Turns out that most of what gets labeled that way isn’t a detection error at all. It’s a true alarm, correctly fired, explained by something legitimate happening in that particular environment, that the analyst waves off anyway because they’ve seen it before. The logic isn’t wrong, it’s just under-contextualized.
That’s exactly what I mean by narrowing a rule instead of turning it off entirely: the alert was right, it just didn’t carry enough context for the analyst to trust it. So, the fix is to create alerts that carry enough context that an analyst trusts the call the first time.
The trade-off nobody likes to say out loud
I won’t pretend this is free. Tightening a rule to catch something subtle, a credential getting misused, movement across a network that shouldn’t be happening, tends to raise your false positive count in the short term.
Loosening a rule to cut volume risks missing exactly that kind of early warning. You don’t get full coverage and a quiet queue from the same rule set at the same time. That trade-off doesn’t go away. You manage which side of it you’re on, you don’t solve it once and move on.
Industry benchmarks give you something to aim for even if they don’t make the trade-off disappear: a false positive rate under ten percent for anything you’re calling actionable, and an alert-to-incident conversion rate above twenty percent for the alerts you’re actually spending investigation time on.
That’s a target, not a guarantee, and it takes the rest of the team telling me what they’re actually seeing so I know what’s repeat activity and what’s genuinely new. Detection engineering done alone, without that loop back to the people working the queue every day, is detection engineering that’s already behind.
I’d also point out that nobody in this field gets to declare victory permanently.
People who’ve watched this industry for decades, longer than I have, make the same observation: there isn’t a single year, across any generation of tooling, where alert fatigue got declared solved.
That’s not a knock on any particular product or platform. It means this isn’t the kind of thing you buy your way out of once. It’s something you manage every week, the same way we do it here, by going back through what got closed and asking whether it should have been.
Where AI actually helps with alert fatigue
This is also where I think the conversation about AI gets misapplied. What it’s actually good for is the layer of judgment right after a rule fires, not replacing the rule and not replacing the analyst.
Once something’s flagged, it should tell us fast whether this is noise we can move past or something that needs a human looking at other data sources immediately. That’s filtering, instead of deciding.
The bottleneck goes beyond headcount, it’s how fast a person can actually investigate something once it lands in front of them. Cut the analysts and hand the same queue to fewer people, and you haven’t reduced anything.
You’ve just made the wait longer before someone reaches the alert that actually matters.
What to ask your security provider about alert fatigue and detection engineering
Here’s what that standard actually produces on our end.
96% of what fires in our environment gets investigated and closed out by my team before it ever reaches a customer’s own people. Instead of being suppressed or hidden from a dashboard, it got looked at, and it turned out to be nothing, which is a different thing entirely from never having looked.
If a security provider’s pitch to you is “we’ve reduced your alert volume,” ask a different question. Ask what their false positive rate actually is, and what happens on the day an alert fires and turns out to mean nothing anyway.
A quiet queue because things are getting hidden isn’t a win. A queue where everything in it earned its place is the only version of this that holds up on the day something real comes through it.
Quick answers
What is alert fatigue? It’s what happens when analysts see the same kind of alert so often that they stop investigating it properly and start closing it out on habit, which opens a visibility gap for the one time it isn’t routine.
Should you reduce alert volume to fix alert fatigue? Not necessarily. Cutting volume by suppressing or disabling noisy rules hides output without fixing what the rule catches. Adjusting the rule’s precision- narrower or broader, depending on what it’s missing- is the fix that actually holds up.
What does high-fidelity detection mean? An alert that fires should always be worth investigating. That standard sometimes means more alerts, not fewer, since a properly tuned rule catches more of what it’s actually supposed to catch.
Where does AI actually help with alert fatigue? In the layer of judgment right after a rule fires, filtering what’s noise and what needs a human, not in replacing the rule itself or the analyst’s judgment on what the alert means.
Written by Brian Edminston, Principal Analyst at OneAxiom
Edited by Julia Araujo, Marketing Manager at OneAxiom
