Runbook Generator by Hand vs With an AI Agent

Writing a runbook by hand means an engineer reconstructs the diagnosis and the fix from memory, chat history and query logs after the incident is over. Using an agent means the runbook is drafted from the diagnosed incident and the fix itself, and a person reviews it before anyone relies on it. We build the second approach as the dw-runbook-generator skill on our Incidents Agent, which generates a reusable remediation runbook from a diagnosed incident and its fix. Below we compare the two so you can decide which parts of the job are worth handing over.

Key Takeaways

• •The hard part of a runbook is not the writing. It is recovering what was actually diagnosed and changed, and that detail decays within days of an incident.
• •An agent that already holds the diagnosis and the fix as structured context can draft a runbook without reconstructing anything from memory.
• •The dw-runbook-generator skill is scoped to one job: turn a diagnosed incident and its fix into a reusable remediation runbook.
• •A human should still approve every runbook before it becomes operational guidance, because a wrong runbook is worse than none.
• •Evaluate any runbook agent on whether it cites the evidence behind each step, not on how polished the prose reads.

Runbook generator by hand vs with an AI agent: the job and who does it

A remediation runbook is the written answer to "this broke before, what do we do when it breaks again?" It names the symptom, the checks that confirm the cause, the fix, and how to verify the fix worked. Google's SRE book argues for a blameless postmortem culture in which incidents produce written learning, and Atlassian's incident management guide describes postmortems as a way to record the cause and the follow-up actions. A runbook is the most operational piece of that output.

In data teams, the person who writes it is usually whoever was on call. That is typically a data engineer who has just spent hours on a failed pipeline, a schema change or a bad backfill, and who has a backlog waiting. Writing the runbook is rarely the most urgent thing on their list, so in our experience it is the first task to slip.

The manual workflow, step by step

This is how the job usually goes by hand. The tools vary by team, but the shape is common.

• •Collect the evidence. The engineer reopens the alert, the orchestrator run history, the warehouse query log and the Slack thread to remember what they actually looked at.
• •Reconstruct the root cause. They write down what the failure was, often from memory, because the exact queries run during the incident are scattered across tools.
• •Record the fix. They note what was changed: a model, a DAG setting, a schema mapping, a permission. The diff may live in a pull request, a hotfix or a console change that never made it to version control.
• •Generalize. They turn a one-off fix into steps that someone else could follow, which means deciding what is specific to this incident and what is reusable.
• •Add verification. They describe how to confirm the fix worked, for example row counts, freshness checks or a downstream test.
• •Publish and link. They put the runbook in a wiki, a repository or an on-call tool, then link it from the alert so the next responder finds it.

Where the manual workflow breaks

The failure points are consistent, and most of them are about information loss rather than writing skill.

• •Memory decay. The longer the gap between the fix and the write-up, the more steps are paraphrased or dropped. Console changes that were never committed are the usual casualties.
• •Handoffs. The person who diagnosed the issue is not always the person who writes the runbook, so the reasoning behind a step gets lost in the transfer.
• •No evidence trail. Runbooks written from memory rarely link back to the queries, lineage or logs that justified each step, so the next reader cannot tell which steps are essential.
• •Inconsistent format. Each author structures runbooks differently, so responders spend time learning the layout during an incident.
• •Silent staleness. A runbook is correct on the day it is written. Schemas and pipelines change, and nothing flags the runbook when they do.

We should be fair to the manual approach here. A senior engineer writing a runbook by hand brings judgment about what matters, what is risky and what the team already knows. That judgment is valuable, and nothing in an agent replaces it. What the agent can replace is the evidence-gathering and drafting around it.

What an agent does instead

The dw-runbook-generator skill takes a diagnosed incident and its fix as input and generates a reusable remediation runbook from them. That scope matters. The skill does not decide what is wrong and does not apply fixes. It works on the output of a diagnosis that already happened, which is why the Incidents Agent is the right place for it: that agent traces root cause across lineage, logs and queries, so the evidence for each step already exists in the same context.

In practice you would run it from the client you already use. Every Data Workers agent is an MCP server, so a developer in Claude Code or Cursor can invoke the capability from the terminal. The Model Context Protocol is the open standard that makes that possible, and Claude Code's documentation covers how it connects to MCP servers. A prompt can be as plain as "turn the incident we just closed into a runbook."

What changes compared with the manual flow is where the inputs come from. The agent starts from the recorded diagnosis and the fix, so it is not asking an engineer to recall them. The generalization step, deciding which parts of the fix are reusable, is still a judgment call, which is why the next section matters.

Manual vs agent: side-by-side comparison

We deliberately avoid putting hours-saved numbers in this table. We do not have a published, attributed benchmark for runbook authoring time, and a made-up figure would not help you decide. The effort difference is qualitative: the agent removes the evidence-gathering and first-draft steps, and leaves the review.

What still needs a human approval

A runbook is an instruction that a future responder will follow under pressure, so a wrong step has real cost. We think these checkpoints should stay with a person:

• •Approving the runbook itself before it is linked from alerts or on-call tools.
• •Confirming the root cause is actually the root cause and not a symptom. An agent can document a misdiagnosis faithfully.
• •Deciding which steps are safe to automate later and which must always involve a person, such as anything that writes to production tables or changes permissions.
• •Removing incident-specific details that should not generalize, for example a one-time backfill range.
• •Retiring or revising the runbook when the pipeline or schema it describes changes.

This matches how we ship the product. The open-source edition is read-only by default, and write actions sit behind approval workflows and audit trails in Pro. The same principle applies to documentation: the agent drafts, a person approves.

How to evaluate an agent for runbook generation

Whether you use ours or another tool, test it on incidents you have already closed, where you know the right answer. We would check five things.

• •Evidence: does each step point to the query, log line or lineage edge that justifies it? A runbook without sources is hard to trust.
• •Fidelity: replay two or three past incidents and compare the generated runbook to what actually fixed them. Count missing steps and invented steps separately.
• •Scope discipline: does the agent stick to documenting the fix, or does it speculate about causes the diagnosis never established?
• •Reusability: could someone who was not on the incident follow the runbook without asking questions?
• •Safety: does it flag steps that write to production, and does it leave approval to a person?

We covered the broader question of how to judge agent output in our posts on incident diagnosis, and our write-up on the Incidents Agent explains how root cause is traced across lineage and logs, which is the input this skill depends on. If your diagnoses are weak, a runbook generator will only make weak diagnoses easier to repeat.

When to stay manual

There are cases where writing by hand is still the better choice. If an incident was novel and the cause is still disputed, wait until the team agrees before generating anything. If the fix involved organizational steps, such as a vendor ticket or a policy exception, the narrative is mostly human and an agent adds little. And if you have a small number of high-stakes runbooks, careful authorship by a senior engineer may be worth the time regardless of tooling.

For the common case, a recurring pipeline failure with a known fix that nobody has time to write down, a drafted runbook that a person reviews in a few minutes is a better outcome than no runbook. Our Pipeline Agent and Schema Agent often supply the context for those fixes, since schema changes and failed DAGs account for many repeat incidents.

Frequently Asked Questions

What is a runbook generator? A runbook generator is a tool that produces a step-by-step remediation guide for a known failure. The dw-runbook-generator skill does this from a diagnosed incident and the fix that resolved it, so the guide reflects what actually happened.

Can an AI agent write a runbook without human review? Technically it can produce a draft, but we do not recommend treating it as final. A runbook directs future actions under pressure, so a person should approve it, especially where steps touch production data or permissions.

Does the agent fix the incident as well? The skill is scoped to generating the runbook from a diagnosed incident and its fix. Diagnosis comes from the Incidents Agent's root-cause tracing, and any fix that writes to your systems follows your approval rules.

How is this different from a postmortem? A postmortem records what happened and why, usually for learning and accountability. A runbook is operational: it tells the next responder what to do. A good team keeps both, and the runbook can be derived from the same incident record.

How do we try it? The OSS edition, dw-claw, is Apache 2.0 and runs locally. Because the agents are MCP servers, you can connect them to Claude Code, Cursor or any MCP-compatible client and run the skill on an incident you have already closed to compare against your own hand-written version.