Skip to content

Alerting

Configure meaningful alerts, escalation policies, and on-call rotations for incident response.

What you'll learn

  • What makes a good alert vs alert fatigue
  • Setting thresholds and SLO-based alerts
  • Escalation policies and on-call rotations
  • Tools: PagerDuty, Opsgenie, Alertmanager

Concept

Good Alerts vs Alert Fatigue

A good alert is actionable — there is something you can do to fix it. It is specific — it tells you what is wrong and where. It is urgent — it matters right now, not tomorrow. A bad alert fires for transient blips, cannot be acted on, or duplicates information. When engineers receive too many bad alerts, they develop alert fatigue — they start ignoring alerts, including the important ones. This is dangerous. Every alert should be reviewed regularly and silenced or removed if it is not useful.

SLO-Based Alerting

Service Level Objectives (SLOs) define your reliability targets — for example, "99.9% of requests succeed within 200ms over 30 days." SLO-based alerts fire when you are at risk of violating the SLO, not when a threshold is crossed. This approach is more principled than threshold-based alerting because it directly measures user impact. Calculate the error budget — the allowed failures within the SLO — and alert when the budget is being consumed too quickly.

Escalation and On-Call

When an alert fires, someone needs to respond. On-call rotations ensure 24/7 coverage by cycling through team members. Escalation policies define what happens if the primary on-call doesn't respond — the alert goes to a secondary, then a manager. Tools like PagerDuty, Opsgenie, and Alertmanager manage these workflows. A good escalation policy gives the on-call engineer enough time to investigate before escalating, but not so much that users suffer.

Alert Routing and Deduplication

In a complex system, many alerts may fire from a single root cause. Deduplication groups related alerts so the on-call engineer gets one notification, not fifty. Routing sends alerts to the right team — database alerts to the DBA team, frontend alerts to the web team. Inhibition suppresses lower-priority alerts when a higher-priority one is active. These features reduce noise and help engineers focus on the actual problem.

Example

              # Prometheus Alertmanager configuration
route:
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'team-default'
  routes:
    - matchers: ['severity="critical"']
      receiver: 'pagerduty-critical'
    - matchers: ['team="database"']
      receiver: 'dba-team'

receivers:
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: 'YOUR_KEY'
  - name: 'team-default'
    slack_configs:
      - channel: '#alerts'
        api_url: 'https://hooks.slack.com/services/...'
            

Alertmanager routes critical alerts to PagerDuty and database alerts to the DBA team. Grouping deduplicates related alerts, and repeat_interval prevents notification spam.

Try it

Common mistakes

The mistake

Alerting on CPU or memory thresholds

The fix

Alert on user-facing symptoms like error rate and latency instead. CPU at 85% might be fine if the system is handling load well. Symptom-based alerts reduce false positives.

The mistake

No deduplication for related alerts

The fix

Configure grouping in Alertmanager so related alerts from a single incident are combined into one notification. Without this, a single outage can generate hundreds of pages.

Practice Alerting

1 exercise

Practice Alerting

Look up unfamiliar terms

A beginner-friendly glossary explains jargon in plain English — variables, functions, DOM, Promise, and more.

View glossary

Build real projects

Apply what you learned by building guided projects with starter code and solutions.

View projects

Decode error messages

Plain-English explanations of common errors — what they mean, why they happen, and how to fix them.

View errors