Alert triage that scales

Route real incidents to a human within minutes, and keep everything else out of their pocket. This playbook sets up a two-tier alerting system with escalation policies and grouped alerts, in about twenty minutes.

The problem: alert fatigue

Most teams start with the default: every monitor notifies every channel. That works at five monitors and collapses at fifty.

When a staging blip sounds exactly like a production outage, people mute the channel. The one alert that mattered gets found an hour later, usually by a customer.

The failure mode is not too many alerts. It is that every alert claims the same urgency, so urgency stops meaning anything.

The principle: separate signal from noise

Decide upfront which failures justify interrupting a person, and which can wait for working hours. For most teams two tiers are enough: Critical for anything customer-facing, Low-priority for the rest.

In Hyperping, a tier is an escalation policy. A monitor without a policy alerts all its configured channels at once, so give every monitor a policy and let the policy decide who hears about it, and when.

The recipe

1. Create a Critical policy for customer-facing services

   Create an escalation policy named Critical and assign it to every monitor a customer can feel: public API, app, checkout, login. We recommend three steps:

   • Step 1, at 0 minutes: Slack message to your #incidents channel.
   • Step 2, after 5 minutes: SMS to the on-call engineer.
   • Step 3, after 10 more minutes: phone call to the on-call engineer.

   Acknowledging the alert stops the escalation, so a responsive team rarely hears a phone ring. Assign the policy from each monitor's Notifications tab.

2. Create a Low-priority policy for everything else

   Create a second policy named Low-priority for internal tools, staging, and batch jobs. Give it a single step: a Slack message to a separate #alerts-low channel. No SMS, no phone calls.

   Nobody gets paged. Sweep the channel once or twice a day during business hours, like a ticket queue.

   If some of these services still deserve a human during work hours, add an email or SMS step that targets an on-call schedule whose rotation has time restrictions covering business hours only.

3. Group alerts for widespread incidents

   When a database dies, thirty monitors go down with it. Without grouping, that is thirty pages for one incident.

   Enable alert grouping on the Critical policy with a short window, 1 to 3 minutes. The first monitor to fail sends the alert, and every other monitor failing inside the window stays silent.

   Grouping can also be set at the project level. When both are configured, the policy setting wins.

Anti-patterns

• Paging everyone for everything. The Everyone option belongs in a last-resort step, not step one. When every alert reaches the whole team, every alert becomes someone else's problem.
• Phone calls as the first step. A 3am call for a blip that self-recovers in 40 seconds burns goodwill fast. Start in Slack, escalate to SMS, save phone calls for the final step.
• Monitors with no policy assigned. They fall back to alerting every configured channel at once, which is exactly the noise you are trying to eliminate. Audit the Notifications tab of monitors created in a hurry.
• One policy for staging and production. If staging can wake you up, your Critical tier is not a tier. Keep non-production monitors on the Low-priority policy.

Next steps