Incident Commander: By Hand vs With an AI Agent

An incident commander coordinates a major incident: they assign workstreams, keep one picture of what is happening, and decide when to escalate or roll back. The Google SRE book chapter on managing incidents splits the response into four roles: incident command, operational work, communications, and planning. By hand, one person holds the coordination thread across all four. With an agent, the coordination and parallel investigation can be automated, while approval for risky actions stays with a human.

Key Takeaways

• •The incident commander role exists to coordinate, not to fix: the SRE book defines four distinct roles so the person in command is not also the person typing commands.
• •By hand, the commander is the integration layer between monitoring, lineage, query logs, and chat. That is where time is lost.
• •Our dw-incident-commander skill, run by the Incidents Agent, coordinates a major incident as commander and runs parallel workstreams to resolution.
• •Production changes, customer-facing communication, and the declaration of severity should still require a named human approval.
• •Evaluate an agent for this job on replayed past incidents, with the audit trail and approval gates inspected first.

The job and who does it today

In a data organization, the commander is usually whoever is on call and senior enough to take charge: a data platform lead, a senior analytics engineer, or an SRE who happens to own the warehouse. The role is temporary. It begins when an incident is declared and ends when service is restored and a postmortem is scheduled.

PagerDuty's open incident response documentation describes the commander as the person with overall responsibility who delegates rather than investigates. That framing matters for data teams, because the temptation is to dive into the failing DAG yourself. The moment the commander starts debugging, nobody is tracking the other workstreams, the stakeholders, or the clock.

Data incidents add a wrinkle that application incidents mostly lack: the blast radius is in the data, not the service. A pipeline can be green while a dashboard serves wrong numbers. The commander has to reason about which downstream tables, models, and consumers are affected, often across tools that were never designed to share context.

The manual workflow, step by step

The manual flow is familiar. An alert fires, or a stakeholder reports a bad number. Someone opens an incident channel and declares severity. The commander then assigns people: one to look at orchestration logs, one to check recent schema or code changes, one to query the warehouse for the actual damage, and one to write status updates for stakeholders.

Each workstream lives in a different tool. Orchestration state is in the scheduler, lineage in the catalog, query history in the warehouse, deploys in git, and conversation in chat. The commander reads each report, mentally merges them, and decides the next move. When two people report conflicting findings, the commander is the one who notices and resolves it.

The resolution phase is where handoffs multiply. A fix gets proposed, someone reviews it, someone applies it, someone reruns the affected jobs, and someone verifies that downstream numbers have recovered. Then comes the postmortem timeline, which is reconstructed from chat scrollback because nobody had time to write things down while the incident was live.

Where the manual approach breaks

The first failure is context fragmentation. The commander is the only place where findings are joined, so the quality of the response is capped by one person's attention. At 2 a.m. with six threads open, that cap is low. Findings get lost, duplicate work happens, and nobody notices a stakeholder has been waiting for an update.

The second failure is serial investigation. Humans can run parallel workstreams only by recruiting more humans, and on-call rotations are small. In our pilot work, incident handling averaged two to four hours of human labor per incident, and that is our own pilot observation rather than an industry benchmark. Most of that time is spent gathering evidence, not deciding.

The third failure is the record. Timelines written after the fact are incomplete, and the postmortem inherits the gaps. If you later need to show an auditor or a customer exactly who approved what and when, reconstructed memory is a weak foundation.

What an agent does instead

The Incidents Agent ships a skill called dw-incident-commander, described as coordinating a major incident as commander and running parallel workstreams to resolution. We keep the claim at that level on purpose. The agent takes the coordination seat, which means it opens the workstreams a human would otherwise assign and keeps the combined picture in one place.

Because every Data Workers agent is an MCP server, the commander runs inside the client your engineers already use, such as Claude Code or Cursor. The Model Context Protocol specification in its 2025-06-18 revision defines how those clients discover and call tools, so the agent is invoked with a prompt like "Why did row counts drop 40% overnight?" rather than through a new dashboard.

The practical difference is that evidence gathering happens in parallel and lands in one thread. Root-cause tracing across lineage, logs, and queries, blast radius questions like "What's the blast radius if I drop this column?", and a proposed fix are all produced for the commander seat to weigh. In early pilot testing, 60% of incidents auto-resolved before human notification, which is our pilot figure and should be read as such.

Manual commander vs agent commander

We should be honest about the last row. A human commander reads a room. They know that finance closes the books on Thursday and that a three-hour delay is tolerable on Tuesday but not this week. An agent can be given that policy, but it will not discover organizational context that nobody wrote down.

What still needs a human approval

Our open-source edition is read-only by default, which means it can investigate and recommend but not change production. Writes arrive in Pro, together with approval workflows and audit trails. We think that split is correct for incident work: diagnosis is cheap to automate and safe to repeat, while mutation is where mistakes become incidents of their own.

We recommend keeping four decisions human. First, declaring or raising severity. Second, any backfill, rollback, or schema change in production. Third, external communication to customers or executives. Fourth, closing the incident and accepting the postmortem. Everything else can run with the agent proposing and a person approving.

The approval itself should be recorded with the evidence the approver saw. If a person approves a rollback, the audit trail should show the root-cause finding and the blast radius they were looking at. That is what makes the decision reviewable later, and it is the part manual processes most often leave out.

How to evaluate an agent for this job

Start with replay. Take three or four real past incidents, including one that was ambiguous and one that was a false alarm, and run the agent against the same evidence the humans had. Compare its root cause and proposed fix with what the postmortem concluded. Do not evaluate on a clean synthetic failure, because the real difficulty is the messy ones.

• •Does it show its evidence for each finding, or only a conclusion?
• •Does it stay read-only until an explicit approval, and can you prove that from the logs?
• •Does it surface conflicting findings instead of choosing one silently?
• •Does the audit trail record who approved which action and when?
• •Does it run in the clients and with the access controls your team already uses?

Then check the failure mode, not the success mode. Give the agent an incident where the data is insufficient and see whether it says so. An agent that guesses confidently is worse than a slow human, because the commander seat is exactly where confident wrong answers do the most damage.

Frequently Asked Questions

Does an AI agent replace the incident commander?

No. It takes over the coordination and evidence-gathering load, while a named person keeps authority over severity, production changes, and communication. We describe it as the agent running the commander seat's workstreams, with a human accountable for the outcomes that carry risk.

What does the dw-incident-commander skill do?

It coordinates a major incident as commander, running parallel workstreams to resolution. It belongs to the Incidents Agent, which also debugs failed pipelines, traces root cause across lineage, logs, and queries, and proposes safe fixes. We do not claim more than that description.

Is it safe to let an agent act during an incident?

Safety comes from scope and gates. The open-source edition is read-only by default. Writes, in Pro, go through approval workflows and are written to a tamper-evident audit trail, so each production change has a recorded approver and the evidence behind the decision.