From Alert Fatigue to Actionable Signals: Designing Better Observability

Alert fatigue is what happens when people receive so many alerts, most of them unimportant, that they stop trusting and responding to them. The fix is not better filtering of the same alerts. It is designing alerts differently: page a person only for urgent, user-facing problems they can act on, give every alert an owner, group related alerts into one incident, and send everything else to tickets or dashboards. Then review alerts regularly and delete the ones nobody acts on.

This article covers what makes an alert actionable, how to choose thresholds, how correlation and ownership reduce noise, and how to keep it from returning. It assumes you already collect telemetry; for the signals themselves, see logs, metrics and traces explained.

What is alert fatigue?

Alert fatigue is the desensitisation that sets in when alerts are frequent, repetitive or rarely require action. The symptoms are familiar:

• alerts acknowledged without investigation, or silenced indefinitely
• a channel full of notifications that nobody reads
• the same alert firing every night and resolving itself
• incidents first reported by customers, even though an alert fired
• on-call shifts that leave people exhausted, and reluctance to join the rota

The danger is not the noise itself. It is that the one alert that matters arrives in a stream people have learned to ignore.

Why does alert fatigue happen?

Most noisy alerting grows through reasonable decisions that were never revisited:

• Alerting on every cause. Teams add an alert for CPU, memory, disk, restarts and queue depth on every component, because each one might indicate a problem. Most of the time, none of them affects users.
• Thresholds copied from defaults. "CPU above 80%" sounds sensible but means different things on different systems, and a busy service may run there happily.
• No duration or hysteresis. A metric briefly crossing a line fires and resolves repeatedly, a pattern called flapping.
• One failure, many alerts. A database outage triggers alerts from every service that depends on it, each sent separately.
• No owner. Alerts go to a shared channel or a central team that cannot fix the underlying service.
• Alerts are added, never removed. Every incident review adds an alert; few ever delete one.

More monitoring makes this worse unless the alerting design changes with it. This is also why AI-powered observability needs care: anomaly detection that flags every unusual pattern can add to the noise rather than reduce it.

What makes an alert actionable?

A useful test is to ask four questions of every alert that can page someone:

These echo the guidance in Google's Site Reliability Engineering book chapter on monitoring, which argues that every page should be actionable, require intelligence to resolve, and relate to a real or imminent problem. The same source suggests that a sustainable on-call load is no more than about two incidents per 12-hour shift, which is a useful benchmark for whether paging volume is reasonable.

Should you alert on symptoms or causes?

Page on symptoms; use causes as context.

A symptom is something users experience: errors, slow responses, failed logins, orders not completing. A cause is a reason that might produce a symptom: a full disk, a saturated connection pool, a restarted container.

Symptom-based alerts are fewer, more stable and directly tied to what matters. They also catch failures nobody anticipated, because they watch the outcome rather than guessing at every route to failure. Causes are still valuable: they belong on dashboards and attached to incidents so the responder can see likely reasons quickly. The disk example shows the exception: a cause that predictably leads to user impact soon enough can justify a page.

How do you set alert thresholds that hold up?

Thresholds are where most noise comes from. A few techniques make them far more reliable.

Base thresholds on objectives, not defaults. A service level objective (SLO) defines the reliability users should get, such as "99.9% of checkout requests succeed over 30 days". The 0.1% that may fail is the error budget. Alerting on the error budget means you alert when users are being let down faster than you have agreed is acceptable, rather than whenever a number looks high.

Alert on burn rate. The burn rate is how fast the error budget is being consumed relative to the rate that would use it exactly over the SLO period. A burn rate of 1 uses the whole budget in 30 days; a burn rate of 14.4 uses 2% of a 30-day budget in one hour. Google's SRE Workbook chapter on alerting on SLOs recommends combining several windows, for example:

The long window ensures enough budget has been spent to matter; the short window confirms the problem is still happening, so the alert resolves quickly once it is fixed. Fast burns page; slow burns become tickets.

Require persistence. A threshold should be breached for a sustained period before firing. Most alerting tools support this; in Prometheus it is the for clause on an alerting rule.

Use hysteresis for flapping signals. Fire when a value crosses one level and clear only when it falls back past a lower one, so a value hovering at the line does not toggle the alert repeatedly.

Be careful with dynamic baselines. Anomaly detection can catch changes static thresholds miss, but "unusual" is not the same as "bad". Use it to add context or to flag issues for review before trusting it to page anyone.

A burn-rate alert in Prometheus might look like this, for a 99.9% objective:

groups:
  - name: checkout-slo
    rules:
      - alert: CheckoutErrorBudgetFastBurn
        expr: |
          (
            sum(rate(http_requests_total{service="checkout",code=~"5.."}[1h]))
            /
            sum(rate(http_requests_total{service="checkout"}[1h]))
          ) > (14.4 * 0.001)
          and
          (
            sum(rate(http_requests_total{service="checkout",code=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{service="checkout"}[5m]))
          ) > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
          team: payments
        annotations:
          summary: "Checkout is failing for users faster than the error budget allows"
          runbook_url: "https://wiki.example.com/runbooks/checkout-errors"

Notice what the rule carries besides the condition: a severity that decides routing, an owning team, a human-readable summary of impact and a runbook link. Those labels and annotations are what make the alert actionable when it arrives. The Prometheus alerting rules documentation covers the syntax.

How does correlation reduce noise?

One failure should produce one incident, not fifty notifications. Several techniques get you there:

• Deduplication. Repeated firings of the same alert update one notification rather than sending new ones.
• Grouping. Alerts sharing a service, cluster or incident are batched into one notification. The Prometheus Alertmanager, for example, groups by chosen labels and waits briefly to collect related alerts before notifying.
• Inhibition. When a higher-level alert is firing, lower-level alerts it explains are suppressed. If a whole data centre is unreachable, there is no value in also paging for every service inside it.
• Dependency awareness. If the database is down, the alerts from services that depend on it are context, not separate problems. Service maps and traces make these relationships visible.
• Shared context. Attaching recent deployments, related cause-level alerts and example traces to the incident means the responder starts with evidence rather than a bare threshold breach.

Correlation depends on the same foundations as good troubleshooting: consistent service names, shared labels and trace context. Those are much easier to achieve with a common instrumentation standard such as OpenTelemetry.

Why does ownership matter so much?

An alert without a clear owner is a notification, not a call to action. Shared channels and central operations teams can triage, but they usually cannot change the code or configuration that caused the problem.

Good ownership practice includes:

• Every alert has an owning team, recorded as a label and routed automatically.
• The owner can act. If the team receiving the page always escalates to another team, route it there directly.
• Owners can change their alerts. The people woken by an alert should be able to tune or delete it.
• Runbooks are owned too. A runbook link that leads to an outdated page is worse than none.
• Ownership follows the service catalogue. When a service changes hands, its alerts move with it.

Ownership also creates the right incentive. Teams that receive their own alerts tend to fix noisy ones quickly.

How do you reduce noise and keep it down?

Cleaning up alerting once helps. Keeping it clean needs a routine.

1. Inventory current alerts. For each, record who receives it, how often it fired over the last few months, and how often anyone took action.
2. Delete or downgrade alerts nobody acted on. If an alert fired often and was always acknowledged without action, it is not a page. Make it a ticket, a dashboard panel or remove it.
3. Classify the rest by urgency: page, ticket or informational. Only pages interrupt people.
4. Add symptom-based SLO alerts for the most important user journeys before removing cause-based pages that currently provide the only coverage.
5. Tune thresholds and durations for alerts that flap.
6. Review after every incident. Ask whether the right alert fired, at the right time, to the right person, with enough context, and whether any alert was noise.
7. Track a few measures over time: pages per on-call shift, the proportion of pages that required action, out-of-hours pages, and incidents first detected by users rather than alerts.

An alert review every few weeks, owned by the people on call, does more than any tool. The aim is a smaller set of alerts that people trust enough to act on immediately.

What does good look like?

Frequently asked questions

What is the main cause of alert fatigue?

Usually, paging on causes rather than symptoms. Alerting on every resource metric across every component produces many alerts that do not reflect user impact, so people learn to ignore them.

What is an SLO burn-rate alert?

It alerts when a service is consuming its error budget faster than its objective allows. Fast burn rates indicate a serious ongoing problem and justify a page; slow burn rates indicate gradual degradation and can be handled as tickets.

How many alerts is too many?

There is no universal number, but Google's SRE guidance suggests no more than about two incidents per 12-hour on-call shift is sustainable. A more practical test is the proportion of pages that needed action. If most did not, there are too many.

Should we delete alerts?

Yes. An alert that consistently fires without anyone needing to act costs attention and trust. Downgrade it to a ticket or dashboard, or remove it, and rely on symptom-based alerts to catch the impact it was meant to warn about.

Conclusion

Alert fatigue is a design problem, not a volume problem to be filtered away. Page only for urgent, user-facing issues someone can act on; base thresholds on objectives and burn rates; collapse related alerts into one incident with context; route each alert to an owner who can fix it; and review the set regularly so noise does not accumulate. The result is fewer pages, faster response and on-call shifts people can sustain. These alerting improvements are some of the easiest to justify in the business case for observability, because their effect on people and incident response is visible quickly.