Alert fatigue from paging on causes instead of symptoms

I get paged for a CPU spike that always fixes itself within a minute. By the third false alarm this week, I've stopped trusting the pager at all.

Intermediate to advancedRead 8 minDoing: an afternoon8 steps

▶ Start

What you'll have at the end

One alerting rule rewritten from a CPU threshold into a two-tier burn-rate rule on the same service's error budget, confirmed silent on the false alarm that started this and confirmed firing on a real outage.

Before you start

You need

A service already answering real traffic, with at least one alert on the pager that fires on an internal resource number like CPU, and nothing yet counting whether a request itself succeeded or failed.

Not covered

Deciding what the team does once this alert's own budget runs low, freeze feature work or keep shipping, is a policy call for elsewhere. This recipe only rewires the alert.

Leans on

8 steps

Where to go next

Your dashboard is missing the four golden signals

for putting this same error rate on a dashboard panel that stays visible all the time, before anything ever pages

Set an error budget and let it decide when to slow down

for deciding what the team does once this alert's own budget starts running low, freeze feature work or keep shipping

Checked 19 Aug 2026

Part of the Backend Observability and Monitoring cookbook