Incident Response for Uptime: Declare, Route, Resolve

Incident response is what a team does between the moment a service breaks and the moment it is working again. For a reliability team that means declaring, assigning roles, mitigating, and only then diagnosing. The order matters: restoring service comes before understanding why, and most of the damage in a bad hour comes from getting that backwards.

What is incident response for a reliability team?

It is a small, rehearsed procedure for an unplanned loss of service. Google's incident management guide frames it as three jobs held by named people, a clear declaration, and a live document everyone writes into. None of that requires tooling you do not already have.

The discipline exists because the default behaviour is worse. Without a procedure, four engineers investigate the same log in silence, nobody talks to customers for forty minutes, and two people restart the same service at once. Every incident response practice below exists to remove one of those failure modes.

When does a blip become an incident?

A single failed check is a blip. Incident response starts at a confirmed, customer-visible loss of service that someone has declared out loud. The gap between those two sentences is where most teams lose time, because nobody wants to be the person who over-reacted.

Customer reports belong in the same funnel. A support ticket saying the site is down is a detection signal, not a declaration, and it needs the same confirmation before anyone pages. The difference is that you already owe that customer an answer, so the clock on the communications side started before yours did.

Make the threshold mechanical so it is not a judgement call at 3am. We prefer two conditions, both required:

• The failure is confirmed from more than one vantage point, not one probe.
• The failure has persisted past a defined number of check cycles.

That second condition has a configuration field. Set alertAfterCycles to 2 on a 60-second check. A fault then has to survive two consecutive cycles before anything pages, which filters a deploy blip without hiding a real outage.

curl -X POST https://uptimeroom.com/api/v1/monitors \
  -H "Authorization: Bearer urm_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "checkout",
    "type": "http",
    "target": "https://example.com/checkout/health",
    "intervalSec": 60,
    "regions": ["london", "los-angeles", "singapore"],
    "expectedStatus": [200],
    "alertAfterCycles": 2
  }'

Three regions rather than two, because two-of-three lets you tell a regional network problem from a service that is down everywhere. The state machine moves UP to SUSPECT to DOWN rather than flipping straight over, and a probe error never counts as a down vote. The reasoning is in cross-region quorum confirmation.

Severity levels that actually change behaviour

A severity scale is only useful if each level implies a different incident response action. Three levels are enough for almost everyone, and a fourth is usually a queue for work that is not an incident at all.

Severity	Meaning	Who wakes up	Customer update
Sev1	Core flow down for most users	On-call plus a second responder	Within 15 minutes
Sev2	Degraded, or one region, or one feature	On-call only	Within 30 minutes
Sev3	Visible but with a workaround	Next working hour	Status page note

Route each level somewhere different. Sev1 goes to a phone, Sev2 goes to a chat channel, Sev3 goes to a ticket queue. If two levels land in the same place, you have two names for one level.

Alert channels are created at /api/v1/channels, documented with the rest of the surface at the API reference. Routing is the cheapest change with the largest effect on an incident response procedure. Sending every severity to one channel is the fastest way to manufacture alert fatigue.

Three roles, even in a team of three

The SRE book's chapter on managing incidents separates three jobs that one person cannot hold at once. In a small team the same human may wear two hats, but naming them out loud still works.

Incident commander. Decides, delegates and keeps the timeline. Does not debug. The moment the commander opens a terminal, nobody is running the incident.

Communications lead. Writes the status-page updates and answers the internal questions so the commander does not have to. What they write and when is covered in incident communication.

Operations lead. The only person changing production. Everyone else proposes changes through them, which is how you avoid two conflicting rollbacks.

Hand the commander role over explicitly when someone has been awake too long. An unspoken handover is how an incident loses its timeline, and a lost timeline is how the review stops being useful.

The incident response timeline

The shape is the same whether the outage lasts four minutes or four hours.

1. Detect. A confirmed monitor state change, or a customer report that you then confirm.
2. Declare. Someone says the word "incident", names a severity and takes the commander role.
3. Mitigate. Roll back, fail over, disable the flag, scale up. Restore service with the cheapest action available.

The first three steps should take single-digit minutes. Everything after them can be done calmly.

4. Communicate. First public update inside the window your severity level promises.
5. Resolve. Confirm recovery from the same vantage points that detected the fault, then say so.
6. Review. Write the postmortem while the timeline is still accurate.

Step three is where teams substitute diagnosis for mitigation. Rolling back an unclear deploy is faster than understanding it, and you can understand it on Thursday. Time spent in each step is what MTTR and MTBF actually measure. Shaving the declare step moves that number further than heroics in the mitigate step ever will.

Closing an incident without reopening it

Recovery needs the same evidence as the declaration. A monitor moving from DOWN to RECOVERING is not the same as a monitor sitting at UP for several consecutive cycles from every region that reported the failure.

We tested the premature-resolution failure mode deliberately on a staging service. Restarting the process turned one region green within forty seconds while a second region was still failing, because a stale connection pool had not yet drained. Anyone watching a single graph would have called it fixed a full two minutes early.

Wait for the confirmation to be as strong as the alarm was. Then post the resolved update, and schedule the follow-up work as normal engineering rather than leaving the incident open as a to-do list.

Common mistakes

Debugging before declaring. Incident response has not started until somebody says it has. Twenty minutes of quiet investigation is twenty minutes customers spent guessing. Declare first. A Sev2 you downgrade in ten minutes costs nothing.

A severity scale nobody can apply. If two engineers would classify the same outage differently, the scale is decorative. Write one example incident per level into the runbook and argue about it once, in daylight.

Confusing the paging rota with the incident response process. There is no on-call schedule or escalation timer in the monitoring layer, so quiet hours, rotations and overrides live in a paging tool. The monitoring side supplies the confirmed trigger and the routing. The human schedule belongs somewhere else.

Treating planned work as an incident. A migration that takes the service down is not an outage. Put it in a maintenance window so the checks suppress and nobody responds, which is the pattern in maintenance windows.

Good incident response is mostly decisions you made in advance. A mechanical declaration threshold, three severity levels routing to three different places, three named roles, and a resolution rule as strict as the alarm. Rehearse the whole thing on a Sev3 so that the Sev1 is boring. Confirmed multi-region detection is the part you can set up today, and three regions per check are available on the Individual plan at $9 a month as of September 2026.