The five tools your on-call engineer juggles at 3:47 AM

Most incident-response post-mortems blame "slow diagnosis" for the bulk of MTTR. After sitting through more war rooms than I'd like to count, I think that's misdiagnosing the diagnosis. The real culprit is rarely thinking. It's navigation.

Here's what actually happens in those first 45 minutes.

3:47 AM. Your phone vibrates. PagerDuty has sent an alert: checkout-svc-prod p95 latency > 800ms, sustained 3 minutes. You roll over, find your laptop, and start the same dance you've done a hundred times.

I want to walk through the next 45 minutes tool by tool, because the tool-switching is the part of incident response we've normalised to the point of invisibility. Once you see it, you can't un-see it.

Tool 1: PagerDuty

You acknowledge the page. PagerDuty shows you the alert, the time it fired, and a single sentence describing the metric breach. It does not show you the underlying graph. It does not show you what changed. It is, in a strict sense, a notification system that happens to have a UI.

Tool 2: Your observability platform

You open the dashboard for checkout-svc. Latency is indeed spiking. You scroll. CPU normal. Memory normal. Request rate normal. Error rate — yes, error rate is up. You click into the breakdown. All 500s from the payment-processor downstream call. You check that dashboard. Its latency is also up. Of course it is. That's why yours is.

But what changed? Anomaly detection helpfully points out that the spike correlates with something twelve minutes ago. Helpfully vague.

Tool 3: GitHub

Recent deploys to the payment-processor repo. There's one from 24 minutes ago. PR #4451, merged by an engineer in a different timezone. You open it. 800 lines of diff across 14 files. The description says "refactor retry logic and update timeout handling."

Could be it. Could not be it. The PR was reviewed and approved. CI passed. The deploy completed successfully. By every signal your tooling cares about, this was a good deploy. And yet.

Tool 4: kubectl

kubectl get pods. kubectl logs. kubectl describe. The logs show retries. Many, many retries. More than would be reasonable. Something in the refactor changed the behaviour.

You're sure now. You also notice some pods in CrashLoopBackOff. The new retry behaviour is OOM-killing them, and the recovery from those crashes is what's driving the cascading latency.

Tool 5: Slack, and the runbook hunt

You ping #incidents. "Looks like PR #4451 to payment-processor caused a retry storm, OOM-killing pods. Considering rollback. Anyone awake?"

While waiting, you go hunting for the rollback runbook. Confluence? Notion? Someone's gist? It exists — you wrote it yourself last quarter — but you can't remember where. You search Slack for "rollback" and find an old thread linking to it.

By 4:25 you've rolled back. Latency recovers within four minutes. You post the resolution and go back to bed.

Of the 38 minutes spent on this incident, about six were spent thinking. The rest was navigating between five tools.

The cost of the juggling

Add up what just happened. Six minutes forming hypotheses, reading code, deciding on the fix. Thirty-two minutes opening tabs, loading dashboards, running kubectl, searching Slack. The cognitive work was small. The friction was enormous.

This is the part that doesn't show up in your MTTR metrics. The clock measures time-to-resolution. It doesn't care whether that time was spent reasoning or context-switching. But your engineers feel the difference. The reason on-call is exhausting isn't the late hour or the alert volume — it's the cognitive cost of orchestrating five tools while half-asleep.

Every additional tool multiplies the friction. Most enterprises have more than five. I've been on calls with teams running eight: Splunk, PagerDuty, Slack, Confluence, Kubernetes, a CI system, a cloud console, and an internal observability platform duplicating half of what the others do. Each tool is good. The juggling is the problem.

What we're actually trying to remove

When I describe Aizen as an autonomous SRE, the resolution piece gets the attention. But for the platform leaders I've been talking to, consolidation matters just as much.

What they want isn't a sixth tool. They want one workflow that knows about the others.

Two things have to be true for that to work. First, the agent does the juggling: when an alert fires, the same system that detected the spike has already read the deploy history, pulled the logs, checked the pods, and found the runbook.

Second — and this is the part that gets underrated — the raw material is all there too. The Splunk query results. The PagerDuty alert and its escalation path. The kubectl describe output. The Slack thread where someone said "seeing this too." The PR diff. Not links out to five tools. The actual content, on one screen.

That second part is what makes the first part trustworthy. An engineer who can't check the agent's reasoning won't let it act. So the evidence has to sit next to the conclusion, not a tab away.

You read one summary. If you want to verify it, everything it was built from is right there. Then you make one decision: let it execute the fix, or take over yourself.

The five-tool dance becomes a single screen.