Airflow Integration for AI Agents: How Data Workers Works Through Airflow's REST API

Airflow decides when and what runs. Data Workers works out why a run failed when the cause sits outside the DAG, and what to run again once it's fixed. This Airflow integration is the wiring between them: Data Workers reads DAG runs and task state through Airflow's REST API, follows a failed task to its cause in dbt, the source or the warehouse, proposes the fix where it lives, drafts the rerun, and after a named engineer approves, checks the run, the table and the deadline downstream.

Your estate probably looks like this. Airflow runs the schedule: a DAG kicks off the Fivetran sync, runs dbt through Cosmos or a dbt job, refreshes a few warehouse tables and tells the BI layer the data is ready. Some teams run Airflow themselves, some on Google Cloud's Managed Service for Apache Airflow (formerly Cloud Composer) or Amazon MWAA, and many on Astronomer's Astro. The current release is Airflow 3.3.2, out since September 17, 2026, and Airflow 2 reached end of life on April 22, 2026, so most estates are on Airflow 3 or moving there.

When a task fails at 3 a.m., Airflow does its job well. It retries, marks downstream tasks upstream_failed, fires a deadline alert and keeps the record of every DAG run. The cause usually lives somewhere else: a retyped column in the source, a dbt model, a warehouse change. That is the part Data Workers takes on, with Airflow staying the orchestrator and the record of what ran.

Key takeaways

• •Airflow schedules; Data Workers finds the cause and proposes the rerun. It reads DAG runs and task state into one context graph through the REST API: /api/v1 on Airflow 2, /api/v2 on Airflow 3.
• •A red task is usually a symptom. Data Workers traces it into dbt, the source or the warehouse, and proposes the fix there as an approval-gated diff for the owner to merge.
• •The rerun goes through Airflow. Data Workers drafts the run and holds the approval. On Airflow 2 Data Workers queues the run through Airflow's REST API; on Airflow 3 the owner triggers the approved run. Data Workers then checks the downstream table and the deadline before it closes the incident.
• •Run it read-only first. You can run Data Workers read-only against Airflow and switch on writes per domain when you're ready.
• •It sits next to Otto, Astro Observe and Google's Managed Airflow Agent. Each works inside its own platform; Data Workers works on Astro, self-managed Airflow and Managed Service for Apache Airflow, and follows the incident into the systems around the DAG.
• •Your coding agent is the way in. Data Workers' agents expose their tools over MCP, so Claude Code, Codex or Cursor can ask about a failed run and hand off the fix.

Going further. For the transformation side of the same loop, read Data Workers + dbt. For the step-by-step path from alerts to agents that fix and verify, read the autonomous data platform playbook.

What Airflow does, and why teams keep it

Airflow is where many data teams run their pipelines, for good reasons. It holds the schedule, the dependencies and the assets that decide what runs and when. Airflow 3 made it a stronger foundation for anything that sits on top: DAG versioning means a run finishes on the code it started with, deadline alerts say when a promise was missed, and the stable /api/v2 REST API exposes runs and task state cleanly. Airflow 3.3 added a task and asset state store, pluggable retry policies, an experimental Dag Results API that waits on a run, and a per-run run_on_latest_version setting that decides whether a cleared, rerun or backfilled run uses the original or the latest DAG code.

Teams keep Airflow because their pipelines are written in it, reviewed in their repository, and understood by the people on call. None of that changes with Data Workers. Airflow stays the scheduler, the executor and the record. Data Workers reads that record, works on the causes that sit outside it, and lines up the right DAG run once the fix is in.

What Data Workers reads from Airflow and writes back

The integration runs over Airflow's REST API. Data Workers detects the API version on its own, so the same connection works on Airflow 2 and Airflow 3, self-managed or managed. On Airflow 3 it authenticates with a bearer token from your auth manager or an Astro API token; on Airflow 2 the usual username and password work too. On a laptop it finds local Astro projects automatically.

The reads go into Data Context Wizard, next to the dbt manifest and the warehouse metadata, so the graph knows which task builds which table and which dashboards wait on it. The writes are deliberately narrow. Clearing task instances, Airflow backfills and pausing DAGs stay in Airflow's UI and your team's hands. Every rerun is a fresh DAG run, proposed with its reason and approved by a person: on Airflow 2 Data Workers queues it, on Airflow 3 the owner triggers it, and Airflow schedules, runs and records it. When the fix is a reload, Data Workers drafts it the same way and reads its status. The undo is written into the plan before the run, and the owner runs it if it's ever needed.

Credit where it's due: Data Workers' Airflow connector reimplements several good ideas from Astronomer's open-source astronomer/agents project (Apache 2.0), including API version detection, the token approach, a read-only switch and local Astro project discovery.

One incident, through Airflow

Here is a scenario most Airflow teams will recognize. It's an illustration, not a customer case.

At 22:10 on Sunday, a source team changes amount in the billing Postgres database from an integer in cents to a decimal string. The billing_daily DAG runs at 03:00 on Monday. Its Fivetran sync task succeeds, and the dbt task that builds stg_billing__invoices fails on a cast error. Every task downstream is marked upstream_failed, and the deadline alert on fct_revenue is set for 06:00. The revenue dashboard in Looker is read at 08:00.

Airflow ran every task, flagged the missed deadline and kept the record of what executed. Data Workers found the cause outside Airflow, proposed the fix there, and after approval specified exactly what to run again. If your team clears the failed run instead, Airflow 3.3's run_on_latest_version setting decides whether it uses the original or the latest DAG code, and that choice stays with your team.

Why doesn't Airflow just do this itself?

Because Airflow is built for one job, and it does it well. Its model ends at the task boundary: a task succeeds, fails, retries or waits for a person. That boundary is what makes Airflow dependable. It doesn't need to know why a Postgres column changed type to schedule ten thousand tasks a day correctly, and it shouldn't have to.

Astronomer has taken the next step inside Astro. Otto investigations, in Labs, start from a failed run and, since September 30, 2026, list up to three ranked suggested fixes: a code patch to one DAG file, an Airflow operation to clear the failed task or run or trigger a new run, or written manual steps, each linked for a person to apply. Their documented context is Airflow, Astro and Observe, which is the right scope for a product built for Airflow and Astro. Fixes in dbt or the warehouse aren't what they describe, and Otto requires Astro. On Google Cloud, the Managed Airflow Agent, available since August 25, 2026, diagnoses failed tasks and DAG runs in the Cloud Console and recommends the steps to fix them.

Writing to production across systems is a different product. It needs blast-radius scoping across every table and dashboard downstream, approvals that name a person, rollback for each change, receipts an auditor can read, context about every other system in the estate, and someone accountable for changes in tools the orchestrator doesn't own. An orchestrator vendor taking on liability for edits to your dbt project and your warehouse would be stepping well outside its job. That cross-system product is the one Data Workers is.

What Airflow and Astronomer cover, as of October 2026

Every row here gives an agent firmer ground: a stable API, deadlines that say when a promise was missed, versioned DAGs, and an MCP ecosystem growing around Airflow. Data Workers builds on all of it.

Otto, Astro Observe, the Managed Airflow Agent and Data Workers: who does which job

Otto is built for Airflow on Astro. It writes DAGs, investigates failures in the context of your deployment, reviews Airflow pull requests and, with investigations, proposes Airflow operations. For engineers changing DAGs on Astro, that's its home turf, and teams who like it should keep it.

Astro Observe tracks data products and SLAs across the DAGs that build them, with lineage from OpenLineage. Its documented setup is for Astro deployments. It answers "which data products are late?"

The Managed Airflow Agent does the same kind of diagnosis on Google Cloud: it investigates a failed task or DAG run from the Cloud Console and returns the problem, the evidence and recommended steps.

Data Workers picks up the question that comes next when the cause isn't in the DAG: what broke, in which system, and is it fixed? It carries the incident into dbt, the source and the warehouse and back to Airflow, with one approval flow and one audit trail. It works the same on Astro, on self-managed Airflow and on Managed Service for Apache Airflow, so teams with more than one Airflow estate get one loop across all of them. All three can run side by side on the same deployment, reading the same REST API.

Airflow MCP and the REST API: how Data Workers connects today

There's no official Apache Airflow MCP server yet; AIP-91, a read-only first phase, sits in the Airflow project's list of AIPs under discussion. The managed platforms have their own: Google's Managed Airflow remote MCP server (GA) and the Astro MCP server (Labs). You don't need any of them for Data Workers. Data Workers talks to Airflow through its REST API, and its agents expose their tools over MCP, so your coding agent (Claude Code, Codex or Cursor) can ask about a failed DAG run, see the cause and approve the proposed fix from the same session. If your team already uses astro-airflow-mcp, it can sit next to Data Workers in the same client. The roles below are an illustration; use what your auth manager or Astro workspace offers.

Week one: read only.

• •Create a service user with a viewer role for Data Workers. That covers DAG runs and task instances.
• •Run Data Workers read-only against Airflow. In read-only mode no agent can trigger anything in Airflow, whatever the token allows.
• •Connect the dbt project and the warehouse read-only too, so the graph ties each task to the tables it builds.
• •Every domain starts at observe. The first thing you see is what each recent failure's cause, fix and rerun would have been.

When you're ready: approved DAG runs.

• •On Airflow 2, add a separate token allowed to trigger DAG runs, limited to named DAGs where your auth manager supports it, and switch on writes for that domain only. On Astro, use a deployment-scoped API token with the narrowest role that allows it.
• •On Airflow 3, the read token is all Data Workers needs: the owner triggers each approved run from its plan, and Data Workers verifies the result.
• •Keep DAG code changes as pull requests in your repository, with no direct deploy rights.

The other direction. Airflow's common.ai provider lets a DAG act as an MCP client. That opens a pattern some teams will want later: a task in your DAG asks Data Workers for context or a check before it publishes a table.

Guardrails

• •Read-only when you want it. Run Data Workers read-only against Airflow, then switch on writes one domain at a time.
• •Autonomy per domain, L0 to L4. On Airflow 2, rerunning a DAG after an approved fix can move to L3 (act, reversibly) while dbt model changes stay at L2 (propose, you approve).
• •Approvals. Each change to production waits for a named human. Human-in-the-loop tasks in your DAGs keep working as they do today.
• •Receipts. Every governed change records what changed, why, who approved it, the blast radius and how to roll it back.
• •Rollback. Code fixes revert like any commit. A DAG run is recorded by Airflow like any other run.
• •No self-approval. No agent can promote its own work.
• •What Airflow owns stays with Airflow. Schedules, retries, deadlines, clears, backfills, DAG versions and auth stay in Airflow, and Data Workers acts only with the grants you give it.

How it fits together

Your team works in its coding agent and reviews in Spellbook Data Catalog, which is in preview. The Data-Agents Swarm does the work, and the Autonomous Data-Conductor runs each incident from detection to verification. Airflow keeps scheduling. Nothing migrates: Data Workers stores metadata and scrubbed facts, not copies of your data, and reaches the rest of the estate through 50+ connectors.

What changes for your team

• •On-call starts from a cause. The engineer who opens a deadline alert finds the failed task, the system it traces to, the proposed fix and the proposed rerun in one place.
• •Fewer blind reruns. A rerun happens after the cause is fixed, so the same task doesn't fail the same way at 07:00.
• •dbt and Airflow owners share one incident. The fix lands in the dbt repo and the rerun in Airflow, linked by one receipt.
• •The DAG authors keep authoring. Whoever writes DAGs today, with or without Otto, keeps doing it.

The fastest first win: failed DAGs with a known rerun

Start with failures that end the same way every time: a late upstream sync, a transient warehouse error, a DAG that needs one rerun once the upstream is fixed. Connect Data Workers read-only for a month and map which task builds which table. Then put those DAGs in propose mode. Each failure arrives in Spellbook with the cause, the proposed fix and rerun, and the blast radius. When your team has approved those proposals as written for a few weeks, open writes for that domain and, on Airflow 2, let Data Workers trigger that class of rerun at L3; on Airflow 3 the owner keeps pressing run on the approved plan.

The case for your CFO

The outcome. Revenue, finance and product numbers arrive on time more often, and when a pipeline breaks, the time from alert to a correct table drops because the cause and the fix are ready when the on-call engineer looks. Fewer cross-system incidents end in a blind rerun that fails again.

The risk story. At L0 and L1, agents only read. At L2 they propose and a named person approves. At L3 they act on changes that can be reversed, inside the domains you open, and L4 is a choice you make per domain, later. Every change carries a receipt with what changed, why, who approved it, the blast radius and the rollback path. Airflow keeps its own run record. Nothing migrates.

Why now. Airflow 3's stable REST API, deadline alerts and versioned DAGs give agents a reliable surface, and MCP has become the standard way coding agents reach it. Orchestration vendors are adding agents inside their own products; the open question for most estates is who owns the incident across the systems around the DAG.

The first win. Failed DAGs with a known rerun, diagnosed and proposed before the deadline alert fires.

What stays the same. Airflow, your DAGs, your auth manager, your dbt project, your on-call rota, and the coding agent your engineers already use as the way in.

The pilot path. Start with a pilot on your own DAGs (pricing); the pilot is credited in full against the first year.

The sentence for upstairs: "Airflow keeps running our pipelines; Data Workers makes sure that when one breaks, the cause gets fixed and the data is right before anyone reads it."

When Airflow on its own is enough

If your DAGs rarely fail, and the cause almost always sits inside the DAG itself, Airflow's retries, retry policies and deadline alerts, plus Otto on Astro, can carry you for now. Once failures start in dbt, the source or the warehouse, or you run more than one Airflow estate, an agent layer that sees the whole estate, fixes the cause and checks the result earns its place.

FAQ

What does the Data Workers Airflow integration connect to? Airflow's REST API: /api/v1 on Airflow 2 and /api/v2 on Airflow 3, detected automatically. It reads DAG runs and task instances, takes task-to-table lineage from the dbt manifest and the team's context-graph notes, and after approval queues new DAG runs on Airflow 2; on Airflow 3 the owner triggers the approved run. It works with self-managed Airflow, Managed Service for Apache Airflow and Astro.

Does Data Workers replace Airflow? No. Airflow schedules and runs the work and keeps the record. Data Workers diagnoses failures whose cause sits outside the DAG, proposes the fix, and drafts the rerun for approval.

Is there an official Airflow MCP server? No. AIP-91, a read-only first phase, is under discussion. Astronomer's open-source astro-airflow-mcp works with Astro and non-Astro Airflow and has writes on unless you set AF_READ_ONLY=true. Data Workers connects over the REST API and exposes its own tools over MCP, so your coding agent can use either or both.

Can Data Workers rerun failed DAGs? Yes, behind a named approval. On Airflow 2, Data Workers queues the new DAG run through the REST API; on Airflow 3, the engineer triggers the run Data Workers drafted. Either way Data Workers then checks the downstream table and the deadline. Clearing task instances and Airflow backfills stay with your team. When the fix is a reload, Data Workers drafts it the same way with the undo written down first, and the owner runs that undo if it's ever needed.

Can we run it read-only? Yes. You can run Data Workers read-only against Airflow, with a viewer-role token, and switch on writes per domain when you're ready.

We already use Otto. Do we need both? Keep Otto if your engineers like it for authoring and investigating DAGs on Astro. Data Workers works beside it on the same REST API and takes the incidents whose cause sits in dbt, the source or the warehouse, on Astro and on any other Airflow you run.

How does this fit with dbt? Most Airflow incidents touch dbt. The fix lands as a dbt diff for the owner to merge and the rerun goes through Airflow; Data Workers + dbt covers the dbt side of the same loop.

Sources

Sources for Airflow and Astronomer capabilities and statuses: Apache Airflow, Astronomer and Google Cloud documentation checked October 3, 2026, including Airflow announcements, Airflow releases (3.3.2, September 17, 2026), Airflow supported versions (Airflow 2 end of life April 22, 2026), Airflow 3.3, Airflow 3.2, upgrading to Airflow 3, REST API security, the Airflow Improvement Proposals index and AIP-91, the common.ai MCP connection, astronomer/agents and its astro-airflow-mcp README, What's new in Otto, Otto investigations, the Otto overview, the Astro MCP server, Astro release notes (September 30, 2026: Otto across Astro, ranked suggested fixes), Managed Service for Apache Airflow release notes (rename April 15, 2026; Managed Airflow Agent and remote MCP server August 25, 2026), the Managed Airflow Agent, Astro Observe GA, Observe and OpenLineage and Cosmos. Product names and statuses change quickly; if we've got something wrong, tell us and we'll fix it.