You're on Managed Service for Apache Airflow (Formerly Cloud Composer): Let Google Run Airflow, and Let Data Workers Run the Operations When a DAG Run's Output Goes Wrong
Google renamed Cloud Composer to Managed Service for Apache Airflow. It runs Airflow for you; Data Workers catches a green DAG run that built the wrong numbers, specifies the approved rerun and verifies the result.
Your team runs Airflow on Google Cloud without running Airflow. On April 15, 2026 Google renamed Cloud Composer to Managed Service for Apache Airflow (formerly Cloud Composer), "Managed Airflow" in the docs. You pick an environment generation (Gen 3 is current; Gen 2 and Legacy Gen 1 are still documented) and an Airflow build: 3.3.1, 3.2.2, 2.11.1 (the default) or 2.10.5 as of the Sept 2 release. Google runs the GKE cluster, schedulers, workers, web server, database and the bucket your DAGs deploy to; your DAGs call BigQuery, Cloud SQL and Vertex AI through Google's operators. Managed Airflow is how Google Cloud teams run Airflow without running Airflow. Data Workers is who runs the operations when a run's output goes wrong: it finds the cause, proposes the fix, routes the rerun for approval and verifies the numbers.
This autumn many of those teams are moving to Airflow 3. Google made Airflow 3 generally available on April 15 and added 3.3.1 builds on Aug 20. Since Aug 26, environments on Airflow 3.3.1 or 3.2.2 can load a snapshot from an Airflow 2.11.1 environment, rolling out region by region, "provided that your DAGs in Airflow 2.11.1 are compatible with Airflow 3". A snapshot moves the DAGs, the configuration and the history. It can't tell you which DAGs now mean something different.
Key takeaways
- •Managed Airflow keeps running Airflow. Environments, DAGs, builds, IAM and the run record stay with Google and your repository. Data Workers connects natively and reads each environment through its Airflow REST API.
- •Green runs get checked against the data. A row-count baseline on the BigQuery partitions your DAGs build catches a successful run that loaded the wrong day.
- •The rerun is exact, approved and verified. Data Workers specifies the run and its date, a named person approves it, and Data Workers follows the run and checks the data afterwards.
- •Fixes arrive as diffs. A DAG change goes to its owner to merge, with the blast radius attached; Data Workers opens a pull request only when your team turns on the GitHub pull-request target.
- •Google's agents and Data Workers split the work cleanly. The Managed Airflow Agent diagnoses failed tasks and DAG runs inside the environment; Data Workers takes the runs that succeeded on the wrong data.
Managed Airflow is how Google Cloud teams run Airflow without running Airflow. Data Workers is who runs the operations when a run's output goes wrong.
Airflow 3 changed one default that matters on every estate with daily DAGs. The Apache Airflow upgrade guide: "The create_cron_data_intervals configuration is now False by default. This means that the CronTriggerTimetable will be used by default instead of the CronDataIntervalTimetable." It applies to DAGs that pass a bare cron string, and the guide warns that templated values like ds "shift between the two timetables". On a trigger timetable, data_interval_start equals data_interval_end: the moment the run fires. A daily DAG whose SQL says {{ ds }} meant yesterday on Airflow 2. On Airflow 3 it means today.
Here is how that plays out at quarter-end. This is an illustration, not a customer case. The DAG orders_daily_load runs on schedule="0 4 *". A BigQueryInsertJobOperator MERGEs orders for {{ ds }} from Cloud SQL for PostgreSQL, through a BigQuery federated EXTERNAL_QUERY, into the partitioned table sales.orders. A KubernetesPodOperator then runs dbt build for the revenue models, including agg_quarterly_revenue, which feeds the Looker "Q3 flash" dashboard. On Wednesday evening the platform team moved to Airflow 3.3.1 by loading a snapshot.
| Time (Thu Oct 1, UTC) | System | What happens |
|---|---|---|
| Wed 19:30 | Managed Airflow | The platform team loads the snapshot of prod-analytics into prod-analytics-af3 (Gen 3, Airflow 3.3.1), checks that every DAG parses and pauses the DAGs in the old environment |
| 04:00 | Managed Airflow | orders_daily_load starts. Its logical date is 2026-10-01 04:00, so {{ ds }} renders as 2026-10-01. On Airflow 2 this run would have loaded Sept 30 |
| 04:06 | BigQuery + Cloud SQL | The MERGE pulls four hours of Oct 1 orders, 7,950 rows, into the Oct 1 partition. The task succeeds. The Sept 30 partition, the last day of the quarter, stays empty |
| 04:14 | dbt on BigQuery | dbt build passes every not_null, unique and accepted_values test. Source freshness is green, because rows landed at 04:06. The DAG run is green |
| 04:15 | Managed Airflow Agent | No task failed, so there is nothing for it to investigate |
| 04:25 | Data Workers + BigQuery | The daily row-count metric for sales.orders partitions flags Sept 30 at zero rows against a weekday baseline of about 48,200, and an Oct 1 partition filling before the day is over. Data Workers opens an incident |
| 04:41 | Data Workers + Managed Airflow | It reads the DAG runs through each environment's Airflow REST API: the last Airflow 2 run had a logical date one day behind its start time; the 04:00 run's logical date equals its start time, with an empty data interval. Cause: the Airflow 3 trigger-timetable default on a bare cron schedule that renders {{ ds }} |
| 04:48 | Data Workers + Slack | Blast radius from lineage: fct_daily_revenue, agg_quarterly_revenue (Q3 short its final day) and the Q3 flash, due to the CFO at 09:00. It proposes a DAG diff and a Sept 30 run, and flags for the platform owner every DAG that pairs a bare cron string with {{ ds }}. The approval request goes to the revenue data owner |
| 07:20 | GitHub + Spellbook | The owner reviews the plan in Spellbook, merges the DAG diff (it pins CronDataIntervalTimetable("0 4 *", timezone="UTC") and takes a load_date param when a run supplies one) and approves the backfill |
| 07:24 | Managed Airflow | The owner triggers orders_daily_load with the conf Data Workers specified, {"load_date": "2026-09-30"}. The undo, deleting the rows that run inserts, is in the plan |
| 07:49 | Managed Airflow + dbt | The run finishes green and dbt rebuilds the revenue models. Data Workers follows the run status to completion |
| 07:55 | Data Workers + BigQuery | It verifies: the Sept 30 partition holds 47,860 rows, back in line with its baseline; the volume checks pass. The receipt lands in Spellbook |
| 09:00 | Looker | The CFO opens the Q3 flash on the full quarter |
Every component did what it was built to do: Google ran the environment, Airflow 3 applied its documented default and dbt's tests passed on the rows they saw. The night needed someone to check the data behind a green run, trace it to a scheduler default, and put the exact rerun in front of a person who could say yes. This domain runs at L2 propose, so the backfill waited for the 07:20 approval.
| Job | What Managed Airflow does | What Data Workers does |
|---|---|---|
| Running Airflow | Runs schedulers, triggerers, workers, web server and database on GKE, with builds up to Airflow 3.3.1 | Works on top of any environment, Gen 2 or Gen 3, Airflow 2 or 3, through its Airflow REST API |
| Upgrading | Loads an Airflow 2.11.1 snapshot into an Airflow 3 environment | Watches the data each migrated DAG builds, so a changed meaning shows up the first night |
| Noticing a problem | Task logs, Cloud Monitoring, and the Managed Airflow Agent on failed tasks and DAG runs | Opens an incident when a green run's output is wrong, with the cause traced across Cloud SQL, BigQuery, dbt and Looker |
| Fixing it | Runs whatever code is in the DAGs bucket | Proposes the DAG change as a diff for its owner to merge, with the blast radius |
| Running it again | Schedules, executes and records the new DAG run | Specifies the run and its parameters, routes it for approval and follows it to completion |
| Proving it | Shows the run as successful | Checks the rebuilt partition and leaves a receipt: cause, approver, run, checks, undo |
Why doesn't Managed Airflow just do this itself?
Because Google built Managed Airflow to run Airflow well: provisioning, scaling, patching, upgrades, networking and security. Its agents work inside that scope, and they are good at it. The Managed Airflow Agent, in the Cloud Console since Aug 25, 2026, "can help you understand, diagnose, and resolve issues with failed Airflow tasks and DAG runs" and optimize an environment, with structured reports that cite log evidence. Gemini Cloud Assist Investigations (Private Preview since April 22) troubleshoots failed task instances and DAG runs. The remote MCP server, GA since Aug 25, lets assistants such as Gemini CLI, ChatGPT and Claude manage environments, list DAGs, trigger DAG runs and fetch task logs, with read-only and read-write OAuth scopes kept apart. Google's Data Agent Kit migrations skill checks and refactors DAG code for Airflow 3.
All of them start from the environment, the code or a failure. In our incident nothing failed, and a bare cron string is valid Airflow 3 code. A scheduler applying a documented default isn't broken, and no log line says the quarter is missing a day. Finding that takes the data side (the BigQuery partitions, the Cloud SQL source, the dbt models, the Looker dashboard), plus a blast radius, a named approver, a recorded rerun, an undo and a check afterwards. That cross-system work is what Data Workers, the agentic data platform, is built to run.
Every tool owns a slice. Data Workers covers the whole lifecycle
Managed Airflow owns its slice deeply: running Airflow on Google Cloud so your team writes DAGs instead of operating clusters. Each point tool adds another console, contract and handoff. Data Workers covers the whole lifecycle with one context, one approval flow and one audit trail, and builds on the environments already running your schedule.
| Stage | Data Workers | Managed Airflow | Why we scored it this way |
|---|---|---|---|
| Catalog & Context | 9 | 3 | The DAG graph and task logs say what ran, and Airflow can emit OpenLineage events. Data Workers keeps one governed context graph of tables, models, owners and lineage across every system. |
| Analytics & Insights | 8 | 1 | Not Managed Airflow's job: the Airflow UI and Cloud Monitoring report on runs and environments, not on the numbers they build. Data Workers answers data questions from governed definitions. |
| Data Quality | 8 | 3 | Checks run as tasks you write; a run is green when its tasks succeed. Data Workers writes and runs checks and baselines on the BigQuery tables the DAGs build. |
| Observability & Incidents | 8.5 | 5 | Task logs, Cloud Monitoring, and the Managed Airflow Agent diagnosing failed tasks and DAG runs. Data Workers diagnoses why a run's output went wrong across systems, proposes the fix behind an approval and verifies it. |
| Pipelines & Ingestion | 8.5 | 9 | Managed Airflow's home stage: Google runs the schedulers, triggerers, workers, web server and database, with Airflow 3 builds up to 3.3.1. Data Workers drafts approved reruns, checks them and keeps what they load right. |
| Schema & Migration | 8 | 2 | Not Managed Airflow's job: snapshots move an environment between Airflow versions, not a changed meaning inside a DAG. Data Workers catches schema and shape changes and assesses blast radius. |
| Governance & Access | 8.5 | 5 | IAM controls who can manage environments and trigger DAGs, and the MCP server splits read-only from read-write scopes. Data Workers routes every data change to a named approver with a receipt. |
| Security & Privacy | 8 | 5 | Private IP, VPC Service Controls, CMEK and Secret Manager protect the environment itself. Data Workers' pull request review flags new columns whose names or annotations look sensitive before the DAGs move them. |
| Cost / FinOps | 8 | 3 | Google's recommendations skill tunes worker and scheduler capacity. Data Workers sums BigQuery spend from the Jobs API and drafts warehouse fixes for their owner. |
| MLOps & Models | 7.5 | 4 | Vertex AI training and scoring pipelines are often DAGs, so Managed Airflow schedules the steps. Data Workers keeps the data under your models healthy. |
For the wider Google Cloud estate, read Data Workers on Google Cloud. Teams running Airflow elsewhere can compare you're on Airflow and you're on Astronomer; for the dbt side of a night like this, see you're on dbt.
How Managed Airflow and Data Workers work together
Engineers ask from their coding agent or Gemini CLI; approvers decide in Spellbook Data Catalog (in preview). Underneath, Data Context Wizard keeps one context graph across Managed Airflow, Cloud SQL, BigQuery, dbt and Looker, the Data-Agents Swarm does the work, and the Autonomous Data-Conductor runs each incident end to end.
What Data Workers reads and triggers. Cloud Composer is a native Data Workers connector, under either name. Each environment serves the Airflow REST API on its web server; Data Workers detects /api/v2 on Airflow 3 or /api/v1 on Airflow 2 and reads DAG runs, their state and the task instances in each run. Task-to-table lineage comes from the dbt manifest and the team's context-graph notes. The one write it supports is a new DAG run with an optional conf payload, queued through trigger_airflow_dag on Airflow 2 environments after a named approval. On Airflow 3 environments the owner triggers every approved run, and a dated backfill like this one is the owner's run on any version, with Data Workers supplying the exact parameters and checking the result. Clears, pauses, environment edits and DAG deploys stay with your team; pin the connection read-only and every write is refused. BigQuery, PostgreSQL (including Cloud SQL), dbt and Looker (read) are native too. The wiring is in the Airflow integration guide.
Setup over MCP today. Every Data Workers agent is an MCP server. The client setup guide documents the path: clone the repo and add one start-agent.sh entry per agent to your client. Google's Managed Airflow MCP server (composer.{region}.rep.googleapis.com/mcp, OAuth with IAM) can sit in the same client; give it the cloudcomposer.readonly scope if environment changes should stay in your platform team's own process.
# Example: Data Workers agents in Claude Code, from a clone of the open-source repo
claude mcp add --scope user dw-incidents -- "$(pwd)/start-agent.sh" dw-incidents
claude mcp add --scope user dw-catalog -- "$(pwd)/start-agent.sh" dw-context-catalog
claude mcp add --scope user dw-quality -- "$(pwd)/start-agent.sh" dw-quality
claude mcp add --scope user dw-connectors -- "$(pwd)/start-agent.sh" dw-connectorsList each agent's tools with your client's own command, such as /mcp. In this story: monitor_metrics and diagnose_incident catch and explain the break; trace_cross_platform_lineage and blast_radius_analysis map what it touches; run_quality_check verifies the rebuilt table; send_slack_alert tells the channel.
Where things run. The agents run in your infrastructure on every tier, for example on GKE in your project, and hold the credentials and model key. Your data stays in your systems; the hosted Conductor sees workflow metadata only. The production remote endpoint serves /mcp and takes an API key or OAuth tokens from your identity provider, such as Okta or Entra ID, verified through JWKS.
One incident, L0 to L4, set per domain:
- •L0 manual. The CFO asks at 09:00 why Q3 revenue looks a day short.
- •L1 observe. The incident opens at 04:25 with cause and blast radius. Nothing changes.
- •L2 propose. Nothing runs until a named person approves; an unanswered request expires and escalates, never auto-grants.
- •L3 act reversibly. Classes with a clean record, such as an undated rerun on an Airflow 2 environment, go ahead without waiting, the undo written first. The DAG diff still waits for its owner.
- •L4 autonomous. A scoped domain handles that class end to end.
More in autonomy levels L0 to L4, how approvals work, is it safe to let AI agents change production data and where does our data go.
What changes for your team
Google keeps running Airflow and your DAG authors keep authoring. What changes is the work after a run: the morning question from finance becomes a diagnosed incident with a proposed fix and an exact rerun waiting for one approval.
The Orchestration agent and Incident Debugging agent posts go deeper. For practice notes, see data engineering on GCP and automating Airflow task failure analysis.
Keep Managed Airflow, or consolidate?
Keep Managed Airflow if you love it; Data Workers works with it from day one. Many teams consolidate once Data Workers runs that slice too.
For nearly every Google Cloud team the answer is keep it: nobody wants to operate Airflow schedulers again. What teams consolidate is the scaffolding around it: row-count tasks pasted into DAGs, a separate freshness monitor, a spreadsheet of "DAGs we checked by hand". Weighing a build on a coding agent plus Google's MCP server? Read build it ourselves with Claude Code and MCP servers: connecting the servers is the easy part; the approvals, undo and verification are the product.
The case for your CFO
The outcome: the numbers your Google Cloud pipelines build arrive right, on time, more often. A DAG run that goes green on the wrong data is caught overnight, fixed behind one approval and checked before the business reads it: in our illustration, a Q3 flash with the full quarter instead of one missing its last day.
The risk story is plain. Agents decide nothing you haven't delegated, and autonomy is set per domain from L0 manual to L4 autonomous. At L2 a named person approves every rerun and fix, and no agent can promote its own work. In Managed Airflow the only write Data Workers supports is a new DAG run, for the domains you open; DAG code changes are diffs your engineers merge. Every change carries a receipt with the cause, approver, run, checks and undo, and an org-wide stop halts all autonomous dispatch. Zero migration: Managed Airflow, your DAGs, BigQuery, dbt and Looker stay as they are.
Why now: Airflow 3 moves are underway this autumn, and a snapshot carries every DAG across, including the ones whose meaning changes. The first win is L1 on one critical chain, such as quarter-close revenue, during the migration. What stays the same: your environments, repository, IAM, review process and on-call rota. For the numbers, see the ROI of agentic data operations, and for the executive view, the Google Cloud guide for data leaders.
The sentence to repeat upstairs: "Google runs our Airflow; Data Workers makes sure a green run means right numbers, and fixes it behind an approval when it doesn't."
Getting started
Start with a pilot. Pick one chain that matters, such as the revenue DAGs behind the quarterly flash, connect Data Workers read-only to that Managed Airflow environment, BigQuery, the source database and the dbt project, and run at L1 so every suspect run arrives with its cause and blast radius. Run it across your Airflow 3 migration and every moved DAG gets checked on its first night. Then turn on L2 for that domain. Plans are on the pricing page, and the pilot is credited in full against the first year.
FAQ
Is Cloud Composer the same product as Managed Service for Apache Airflow? Yes. Google renamed Cloud Composer to Managed Service for Apache Airflow on April 15, 2026. URLs, APIs and parts of the documentation still say Composer, and Data Workers' Cloud Composer connector works with it under either name.
What exactly does Data Workers read from Managed Airflow, and what does it trigger? It reads DAG runs, their state and the task instances in each run through the environment's Airflow REST API (/api/v2 or /api/v1, detected automatically). On Airflow 2 environments it can queue a new DAG run, with an optional conf payload, after a named approval; on Airflow 3 the owner triggers the approved run. Dated backfills, clears, pauses, environment changes and DAG deploys stay with your team, with Data Workers specifying the run and checking the result.
We use the Managed Airflow Agent. Do we need both? Keep it for failed tasks, failed DAG runs and tuning the environment. Data Workers works beside it on the same environment and takes the incidents whose cause or impact sits outside the DAG, including green runs with wrong data.
Can Data Workers help with our Airflow 2 to Airflow 3 migration? It watches the data each migrated DAG builds, so a DAG whose dates or volumes shift after the move shows up as an incident with its cause, such as the cron timetable default, and a proposed diff. The migration itself, snapshots and environment changes, stays with your platform team and Google's tooling.
Can a backfill make things worse? At L2 every rerun waits for approval, and the plan records the undo before it runs; the owner runs the undo if it's needed. Data Workers checks the rebuilt tables afterwards and escalates a failed check to a person.
Sources
- •Google Cloud, Managed Service for Apache Airflow release notes (rename and Airflow 3 GA, Apr 15, 2026; Gemini Cloud Assist Investigations, Apr 22; Airflow 3.3.1, Aug 20; Managed Airflow Agent and MCP server GA, Aug 25; snapshot migration, Aug 26; current builds, Sept 2), https://docs.cloud.google.com/composer/docs/release-notes (checked Oct 3, 2026)
- •Google Cloud, Managed Airflow overview (generations, components, Private IP, VPC Service Controls, CMEK; updated Sept 30, 2026), https://docs.cloud.google.com/composer/docs/composer-3/composer-overview (checked Oct 3, 2026)
- •Google Cloud, Agentic capabilities in Managed Airflow (agent, MCP tools, Data Agent Kit skills; updated Sept 30, 2026), https://docs.cloud.google.com/composer/docs/composer-3/agentic-capabilities-overview (checked Oct 3, 2026)
- •Google Cloud, Use the Managed Airflow remote MCP server (endpoint, OAuth scopes; updated Oct 1, 2026), https://docs.cloud.google.com/composer/docs/composer-3/use-composer-mcp (checked Oct 3, 2026)
- •Apache Airflow docs, Upgrading to Airflow 3 (
create_cron_data_intervalsdefault), https://airflow.apache.org/docs/apache-airflow/stable/installation/upgrading_to_airflow3.html (checked Oct 3, 2026) - •Apache Airflow docs, Timetables (trigger timetables), https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/timetable.html (checked Oct 3, 2026)
- •Data Workers open-source repository (tool registrations), https://github.com/DataWorkersProject/dataworkers-claw-community (checked Oct 3, 2026)
- •Data Workers client setup guide, https://dataworkers.io/opensource-docs/client-setup/ (checked Oct 3, 2026)