How to Build Self-Healing Dagster or Airflow Pipelines That Open a PR When a Step Fails

To build a self-healing Dagster or Airflow pipeline, attach a failure trigger (a Dagster failure_hook or an Airflow on_failure_callback) that sends the failure context to an agent, and let that agent propose its fix as a pull request rather than a change to production. Paradime's self-healing Dagster guide reports that a typical healing run takes 2 to 6 minutes from step failure to open PR. That is a vendor-reported figure, not an independent benchmark.

Key Takeaways

• •Dagster and Airflow both expose a failure trigger you can point at an agent: a failure hook in Dagster, and on_failure_callback in Airflow.
• •Paradime's guide reports (vendor-stated) that a healing run takes 2 to 6 minutes from step failure to open PR.
• •The safe output of a healing agent is a pull request, never a direct write to production code or data.
• •Guardrails matter more than the trigger: a no-op when configuration is missing, non-blocking dispatch, a size cap on the failure message, limited tools, one PR per run.
• •In Dagster, materializing straight from the asset graph bypasses hooks, so launch through a job that carries the hook.

What a self-healing pipeline actually is

A self-healing pipeline is not one that fixes itself invisibly. It is a pipeline where the time between a failure and a reviewable fix is short, because a program collected the evidence, diagnosed the cause and drafted the change before an engineer opened a laptop. The human still decides. What changes is that the human starts from a diff instead of a stack trace.

We make that distinction early because the word self-healing invites the wrong design. If an agent can edit production DAGs or rerun backfills on its own authority, a bad diagnosis becomes an outage. If an agent can only open a pull request, a bad diagnosis becomes a closed PR and a few seconds of reading.

Detection is the easy half of this. Most teams already have alerts. The expensive half is the gap between the alert and the fix, which is filled by an engineer pulling logs, reading the failing model or task, and writing a patch. That gap is what the agent shortens. Our earlier posts on self-healing Airflow with an AI agent and DAG recovery step by step cover the recovery side; this post focuses on the wiring and the PR guardrails.

The five-layer structure

We structure the pattern as five layers. Each layer has one job, and each can be inspected on its own, which is what makes the whole thing auditable. The layers are the same for Dagster and Airflow; only the first one differs.

Layer 1: wiring the trigger in Dagster

Dagster documents a failure_hook decorator that creates a hook on step failure events. The hook receives a context from which you can read the op name and the thrown exception through op_exception, and it reaches resources through context.resources, so any resource it uses must be declared in required_resource_keys. That context is the raw material for the agent payload.

Dagster also offers run status sensors, which react to run-level status changes. Hooks fire on step failures during op execution; sensors observe the run after the fact. We prefer the hook when you want the failing step's exception, and a sensor when you want one trigger covering many jobs without editing each of them.

One trap is worth stating plainly. Paradime's guide says that materializing directly from the asset graph runs Dagster's internal __ASSET_JOB, which carries no hooks. If your team launches assets from the UI asset graph, a hook attached to a named job never fires. Launch through a job that has the hook attached, or use a sensor, and test the failure path deliberately before you trust it.

Layer 1: wiring the trigger in Airflow

In Airflow, on_failure_callback is available on both DAGs and tasks. Since Airflow 2.6.0 it accepts a list of functions, so you can keep your existing alerting callback and add the agent dispatch beside it. Every callback receives a context mapping with runtime information about the task instance, including identifiers such as the run ID.

The Airflow docs list several caveats that shape the design. Callbacks run only when a worker's execution changes the state, so state changes made through the CLI or UI do not trigger them. Errors raised inside a callback are written to dag processor logs rather than task logs, which are not shown in the UI by default. And for DAG-level callbacks, the context describes one chosen task instance, so the docs advise against relying on it except for human analysis.

The practical consequence: put the callback on the task when you need the failing task's details, and keep the callback body tiny. It should gather identifiers, truncate, send, and return. Anything that can raise should be wrapped so the callback itself never becomes a second failure. Retry behavior is a separate hook: on_retry_callback is its own event type, so confirm in a test DAG which event fires on your retry settings before you wire the agent.

Layer 2: dispatch that cannot make things worse

The dispatch layer is where most of the safety lives, and the Paradime guide is a useful reference for its shape. It describes a hook that logs a warning and returns if its environment variables are unset, so the original op failure still surfaces and local runs do not fail because of the hook. It dispatches fire-and-forget so the run's teardown is not held up by the agent session, and it truncates the failure message at 12,000 characters.

We would add three rules of our own. First, never send secrets: strip connection strings and tokens from the payload before it leaves the process. Second, deduplicate by run ID, so a flapping task does not open ten pull requests. Third, rate-limit per pipeline, so a systemic outage produces one investigation rather than hundreds. These are our recommendations, not features of either orchestrator.

Layers 3 to 5: triage, patch and review

Once the agent has the payload, the order of operations matters. The Paradime guide has the agent post a triage summary before touching any code, put all fixes for a failed run on one branch and in one PR with each error documented separately, and prefer minimal changes over rewrites. We think that ordering is right: diagnosis first, then the smallest patch, then one artifact for a human to review.

The tool set should be narrow. The same guide limits the agent to reading files, searching files and text, running SQL queries and running terminal commands. We would go further for a first deployment: read-only SQL, no write credentials to the warehouse, and a git token scoped to opening branches and pull requests. GitHub's REST API supports a draft parameter when creating a pull request, which lets the agent mark its work as not ready to merge until a person promotes it.

On our side, the Pipeline Agent and the Orchestration Agent are built around this division of labor: diagnose, propose, hand off. Each agent is an MCP server, and the MCP specification defines the tool interface a client uses to call them. Whatever agent you use, the contract is the same: it may read broadly, and it may write only to a branch.

Approval tiers: what the agent may do alone

Not every fix deserves the same scrutiny. The table below is a recommendation we use for deciding how much autonomy a class of fix gets. It is a policy proposal, not a measured result, and you should adjust it to your own risk tolerance.

The key line is between code and data. A pull request is reversible and reviewable. A backfill that overwrites a table is neither unless you have bounded it. For that reason we keep data repair at propose-only until the team has a record of good proposals; our post on bounded backfills describes how to limit the range.

Audit trail and rollout

Every agent action should be reconstructable: the payload it received, the files it read, the queries it ran, the diff it produced, and who approved it. The MCP protocol itself does not give you that, which we cover in what a data agent audit trail must record. Treat the pull request as one record and the agent's tool log as the other.

For rollout, we suggest a staged path. This is a suggested plan, not a measured timeline. In week one, run the hook in shadow mode on a single non-critical pipeline: the agent writes its triage to a log and opens nothing. In week two, let it open draft PRs on that pipeline and measure how many a human would have merged unchanged. In weeks three and four, extend to more pipelines and only then consider tiers that carry less review.

Frequently Asked Questions

Should the agent merge its own pull request?

We recommend not, at least initially. The value of the pattern is a short path to a reviewable fix, not the removal of review. If you later allow auto-merge for the lowest-risk class, such as a documented config typo, require passing tests and keep the draft PR record.

Does on_failure_callback fire on every retry?

The Airflow callbacks page does not state this explicitly. It lists on_retry_callback as a separate event invoked when a task is up for retry, so test your own retry configuration with a deliberately failing task before relying on either callback.

Why did my Dagster hook never fire?

The most common cause is launching through the asset graph. Paradime's guide notes that this path runs the internal __ASSET_JOB, which carries no hooks. Attach the hook to a named job and launch that job, or use a run status sensor.

Can the Data Workers agents do this read-only?

Yes. Each Data Workers agent is an MCP server you can call from Claude Code, Cursor or another MCP client, and the open-source edition is read-only by default, so it can diagnose a failure without write access. See dataworkers.io for the current edition details.