Dagster and Prefect Integration for AI Agents: How Data Workers Works Through Your Runs and Flows
A Dagster and Prefect integration for AI agents: Data Workers reads failed runs and flow runs, fixes the cause in dbt or the source, and queues approved runs.
Dagster and Prefect run the work and keep the record of every run. Data Workers works out why a run failed when the cause sits outside your Dagster code location or your Prefect flow, fixes it in the system it came from, and hands back one approved run. This page covers the Dagster integration and the Prefect integration together: Data Workers reads Dagster runs and their steps through the GraphQL API, reads Prefect flow runs and task runs through the REST API, proposes the fix where it lives, and after a named engineer approves, queues a run of the Dagster job or a flow run from the Prefect deployment and checks the data downstream.
Many estates now run both. Analytics engineers model the warehouse as Dagster assets, with dbt models loaded as assets and asset checks guarding the tables that matter. Platform and ML teams run Prefect flows from deployments on work pools, with retries and automations around them. On July 13, 2026, Prefect announced it is acquiring Dagster Labs. Dagster's site now says "Dagster is now part of Prefect", and Prefect's FAQ says it "acquired Dagster Labs in July 2026". Both companies say the same thing about the product: Dagster keeps its name and open source license, Dagster+ remains a supported commercial offering, and customers have no migration or action to take. Dagster 1.13.25 shipped on October 1 and Prefect 3.8.7 on September 26.
When a run fails at 2 a.m., both orchestrators do their job well. Dagster marks the run FAILURE, shows which step failed and holds back downstream assets when a blocking asset check fails. Prefect moves the flow run to Failed or Crashed, retries where you told it to and fires your automations. The cause usually lives somewhere else: a source change, a dbt model, a warehouse table. That is the part Data Workers takes on, with Dagster and Prefect staying the orchestrators and the record of what ran.
Key takeaways
- •Dagster and Prefect run the work; Data Workers finds the cause and proposes the rerun. It reads Dagster run and step state over GraphQL and Prefect flow run and task run state over REST, on Dagster+ or Dagster OSS and on Prefect Cloud or self-hosted Prefect.
- •One cause, one incident. When a Dagster asset check and a Prefect flow fail for the same reason, Data Workers links them and proposes one fix, as an approval-gated diff for the owner to merge.
- •The reruns go through the orchestrators. After approval, Data Workers queues a run of the named Dagster job, then a Prefect flow run from its deployment, and checks the tables before it closes the incident.
- •Approval sits outside the actor. The Dagster+ MCP server can launch runs and edit alert policies with the caller's permissions. Data Workers keeps approval with a named person in a separate plane, and no agent can promote its own work.
- •Run it read-only first. Connect read-only and open writes per domain when you're ready.
What Dagster and Prefect do, and why teams keep them
Dagster is built around assets. Teams declare the tables, models and files they care about, and Dagster works out what to materialize, when, and in what order. Asset checks put tests next to the data they guard, and a blocking check stops bad data from flowing downstream. Dagster+ adds hosted deployments, branch deployments, alert policies, Insights and an audit log.
Prefect is built around flows: plain Python functions that become observable, retryable workflows, run from deployments on work pools. Its state model is rich (Late, AwaitingRetry, Paused for a manual approval, Crashed for an infrastructure failure), and its automations react to events anywhere in the stack.
Teams keep both because their pipelines are written in them and understood by the people on call. Dagster and Prefect stay the schedulers, the executors and the record. Data Workers reads that record and asks each orchestrator to run again once the fix is in.
What Data Workers reads and writes back
The Dagster connection runs over Dagster's GraphQL API, against the OSS webserver or a Dagster+ deployment using a Dagster+ API token. The Prefect connection runs over Prefect's REST API, against Prefect Cloud with an API key or against your own Prefect server.
| Data Workers reads | Data Workers writes back, after approval |
|---|---|
Dagster run status, from QUEUED to SUCCESS or FAILURE | A Dagster run of a named job, queued through launchRun with trigger_dagster_job |
| Dagster step stats: which step in the run failed | A Prefect flow run from a named deployment, queued through create_flow_run with trigger_prefect_flow |
Prefect flow run state: COMPLETED, FAILED, CRASHED and the rest | The fix for the cause, in dbt or the warehouse, as a diff for the owner to merge |
| Prefect task runs: per-task state and run count | Only for the domains you open for writes |
The reads go into Data Context Wizard, next to the dbt manifest and the warehouse metadata, so a failed step or task lines up with the dbt model and table it builds and the dashboards and features that wait on it. The writes are deliberately narrow. Terminating runs, partition backfills, re-execution from a failed step, Prefect state changes and alert policies stay in Dagster's and Prefect's hands and yours. When a fix needs a backfill, Data Workers writes the plan (which partitions, which assets, in what order) and the owner launches it. The pipelines agent can also generate new pipelines as Dagster jobs or Prefect flows, so a team that adds a pipeline gets it in the orchestrator it already runs.
One incident, through Dagster and Prefect
Here is a scenario many teams running both tools will recognize. It's an illustration, not a customer case.
At 21:40 on Sunday, the app team changes discount in the orders Postgres database from a percentage (0 to 100) to a fraction (0 to 1). Fivetran lands the new values overnight. In Dagster, daily_orders_job materializes the dbt assets on Snowflake at 02:00, and a blocking asset check on fct_orders, discount_in_range, fails at 02:20. In Prefect, the churn_features/daily deployment runs at 04:00 and feeds the churn model's feature table. The finance dashboard and the churn scores are read at 08:30.
| Step | Where it runs | Handoff |
|---|---|---|
| 1. Detect | Dagster GraphQL API | Data Workers reads the failed run of daily_orders_job and its step stats: the fct_orders step failed, the assets after it did not materialize. |
| 2. Diagnose | dbt and Snowflake | Data Workers compares the landed discount values with the last 30 days and the source definition in the dbt manifest. Every row since the 21:40 change sits between 0 and 1. The job and the check are fine; the source changed meaning. |
| 3. Propose | GitHub and dbt CI | With the GitHub pull-request target on, Data Workers opens an approval-gated dbt pull request that rescales discount in the staging model and tightens the source test, with the blast radius attached: stg_orders, fct_orders, two finance dashboards and the Prefect churn_features/daily deployment. Your dbt CI runs the tests. |
| 4. Link | Prefect REST API | At 04:05 the churn_features flow run fails its freshness task because fct_orders didn't update. Data Workers reads the flow run and its task runs and attaches them to the same incident, so Prefect's owners see one cause instead of a second mystery. |
| 5. Approve | Spellbook or GitHub | At 07:10 the on-call engineer reviews the diff and the two proposed runs, approves and merges. Nothing changes in production before that. |
| 6. Rerun in Dagster | Dagster GraphQL API | Data Workers queues a run of daily_orders_job with trigger_dagster_job. Dagster runs it, the asset check passes, and the run records SUCCESS like any other. |
| 7. Rerun in Prefect | Prefect REST API | Once the Dagster run succeeds, Data Workers queues a new flow run from churn_features/daily with trigger_prefect_flow. Prefect schedules it on the work pool and records Completed. |
| 8. Verify and close | Snowflake and Spellbook | Data Workers checks the discount distribution and revenue totals against the prior week and the churn feature row counts, then closes the incident with a receipt: the cause, the PR, the approver, both runs and the checks that passed. |
Dagster's asset check did exactly what it should: it stopped bad numbers from reaching fct_orders. Prefect's freshness task did the same for the churn features. Data Workers found the cause outside both, proposed the fix there, and after approval told each orchestrator exactly what to run, in the right order.
Why doesn't Dagster or Prefect just do this itself?
Because each is built for one job, and does it well. An orchestrator's model ends at the run: a step succeeds or fails, a check passes or blocks, a flow run retries or crashes. That boundary is what makes them dependable.
Both are adding AI inside their own products, scoped to that job. Dagster+ AI, in early access preview, scans a deployment for failure patterns, creates Issues that summarize problems and suggest solutions, and can dispatch an agent from an Issue to create a fix as a pull request. Its documented context is the deployment's failure patterns and run logs. Prefect Cloud's AI log summaries turn a flow run's logs into one line. That is the right scope for an orchestrator vendor.
Writing to production across systems is a different product. It needs blast-radius scoping across every table, dashboard and downstream flow, approvals that name a person, rollback for each change, receipts an auditor can read, context about every other system in the estate, and someone accountable for changes in tools the orchestrator doesn't own. An orchestrator vendor taking on liability for edits to your source contracts and your warehouse would be stepping well outside its job. That cross-system product is the one Data Workers is.
Dagster+ MCP, Prefect MCP and why approval sits outside the actor
Both vendors now give coding agents a way in, and both are worth using.
| Surface | What it does | Status (Oct 2026) |
|---|---|---|
| Dagster+ MCP server | Remote server for Dagster+. View, launch and delete runs; view and launch assets; read run logs; create, update and delete alert policies; manage Dagster+ Issues and Insights metrics. OAuth with your permissions, or service-user headers | Beta, Dagster+ only |
| Dagster skills | dagster-expert skill in dagster-io/skills for dg CLI usage, asset patterns and automation, installed through the dagster plugin (which replaced the dagster-expert plugin on September 14, 2026) | Available, Apache 2.0 |
| Dagster+ AI and Issues | Proactive monitoring creates Issues with likely root causes; an Issue can dispatch an agent to open a fix PR | Early access preview |
| Prefect MCP server | Hosted (Prefect Cloud OAuth) or local. Inspects deployments, flow runs, task runs, work pools and logs, and searches Prefect docs. Read-only by design | Beta |
Prefect workflows skill | Ships with the Prefect plugin for Claude Code and Codex; sends mutations to a separately authenticated Prefect CLI, only when the user asks | Available |
| Prefect AI log summaries | One-line summary of a flow run's logs, also in automation notifications | Available in Prefect Cloud |
Look at the Dagster+ MCP permissions table. An agent holding the same rights can launch a run, delete a run and also create, update or delete the alert policy that would page someone if that run went wrong. For an engineer at the keyboard, approving each tool call, that's a fast way to work. For an agent working on its own overnight, it means the actor can change production and change the alarm that watches production. Dagster's docs recommend a service user with its own headers when you want different MCP permissions, and Prefect's docs warn that a client with terminal access can run destructive CLI commands outside the read-only server. Both point at the same principle: the thing that acts shouldn't also hold the approval and the alarm.
That is how Data Workers is set up. Approval lives in a separate plane, with a named person in Spellbook Data Catalog (in preview) or on the pull request. No agent can promote its own work. Data Workers takes a narrow grant in each orchestrator, enough to launch named jobs and create runs from named deployments, and alert policies, automations and RBAC stay with your team. Your Dagster+ MCP and Prefect MCP sessions can sit next to Data Workers in the same coding agent; each does its own job.
How Data Workers connects today
The roles below are an illustration; use what your Dagster+ RBAC and Prefect workspace roles offer.
Week one: read only.
- •Dagster: create a Dagster+ service user with a viewer role and an API token, or point Data Workers at your OSS webserver's GraphQL endpoint.
- •Prefect: on Prefect Cloud, create a service account with the narrowest workspace role your plan offers. On self-hosted Prefect, use your server URL.
- •Run Data Workers read-only. In read-only mode no agent can launch or create anything in either orchestrator, whatever the token allows.
- •Connect the dbt project and the warehouse read-only too, so each step and task ties to the tables it builds.
Each connection takes two things: the endpoint (your Dagster+ deployment URL or OSS webserver, and your Prefect Cloud workspace API URL or server URL) and the token or key above. With read-only on, the run tools aren't registered at all.
When you're ready: approved runs.
- •Give the Dagster service user a role that can launch runs in the deployment you name, and switch on writes for that domain only.
- •Give the Prefect service account the right to create flow runs on the deployments you name.
- •Keep code changes as pull requests in your repositories, with no direct deploy rights.
Guardrails
- •Read-only when you want it. Run Data Workers read-only against both orchestrators, then switch on writes one domain at a time.
- •Autonomy per domain, L0 to L4. Rerunning a job after an approved fix can move to L3 (act, reversibly) while dbt model changes stay at L2 (propose, you approve).
- •Approvals. Each change to production waits for a named human. Prefect's own
Pausedapprovals inside your flows keep working as they do today. - •Receipts. Every governed change records what changed, why, who approved it, the blast radius and how to roll it back.
- •Rollback. Code fixes revert like any commit. Each run is recorded by Dagster or Prefect like any other run.
- •No self-approval. No agent can promote its own work.
- •What the orchestrators own stays with them. Schedules, sensors, declarative automation, asset checks, retries, automations, alert policies, RBAC and code locations stay in Dagster and Prefect.
How it fits together
Your team works in its coding agent and reviews in Spellbook. The Data-Agents Swarm does the work, and the Autonomous Data-Conductor runs each incident from detection to verification. Dagster and Prefect keep orchestrating. Nothing migrates: Data Workers stores metadata and scrubbed facts, not copies of your data. The same loop runs through Airflow if part of your estate still lives there; Data Workers + Airflow covers it, and Data Workers + dbt covers the dbt side of every fix on this page.
What changes for your team
- •On-call starts from a cause. The engineer who opens a failed Dagster run or a crashed Prefect flow finds the cause, the fix and the proposed runs in one place.
- •Two teams, one incident. The analytics team on Dagster and the ML or platform team on Prefect see the same incident and the same receipt.
- •Reruns in the right order. The Prefect flow runs after the Dagster job succeeds, so the same failure doesn't repeat downstream.
The fastest first win: failures that end in one rerun
Start with failures that end the same way every time: a late upstream sync, a transient warehouse error, a job or flow that needs one rerun once the upstream is fixed. Connect Data Workers read-only for a month and map which steps and tasks build which tables. Then put those jobs and deployments in propose mode. Each failure arrives in Spellbook with the cause, the proposed fix and runs, and the blast radius. When your team has approved those proposals as written for a few weeks, open writes for that domain and let Data Workers queue that class of run at L3.
The case for your CFO
The outcome. Finance, product and model numbers arrive on time more often, and when a pipeline breaks, the time from failure to a correct table drops because the cause and the fix are ready when the on-call engineer looks. One cause produces one incident across both orchestrators, instead of two teams chasing the same problem.
The risk story. At L0 and L1, agents only read. At L2 they propose and a named person approves. At L3 they act on changes that can be reversed, inside the domains you open, and L4 is a choice you make per domain, later. Approval sits outside the agent that acts, and alert policies stay with your team. Every change carries a receipt with what changed, why, who approved it, the blast radius and the rollback path. Dagster and Prefect keep their own run records. Nothing migrates.
Why now. Dagster and Prefect are now one company, both with MCP servers in beta and AI features inside their products. Coding agents can reach your orchestrators today. The open question is who owns the incident across the systems around them, and who approves what an agent does.
The first win. Failures that end in one rerun, diagnosed and proposed before the morning.
What stays the same. Dagster, Prefect, your assets and flows, your RBAC, your dbt project, your on-call rota, and the coding agent your engineers already use.
The pilot path. Start with a pilot on your own jobs and deployments (pricing); the pilot is credited in full against the first year.
The sentence for upstairs: "Dagster and Prefect keep running our pipelines; Data Workers makes sure that when a run fails, the cause gets fixed where it started and both finish the day with the right data."
FAQ
What does the Data Workers Dagster integration connect to? Dagster's GraphQL API, on the OSS webserver or a Dagster+ deployment with an API token. It reads run status and step stats, and after approval queues runs of named jobs.
What does the Prefect integration connect to? Prefect's REST API, on Prefect Cloud with an API key or on a self-hosted server. It reads flow runs and task runs, and after approval queues flow runs from named deployments.
We already use the Dagster+ MCP server. Do we need both? Keep it for engineers working in their coding agent. Data Workers works beside it and takes the incidents whose cause sits in dbt, the source or the warehouse, with approval held by a named person and a receipt for each change.
The Prefect MCP server is read-only. How does Data Workers write? Through Prefect's REST API, with a service account you scope, and only after approval. Prefect's MCP server stays read-only and can run in the same client.
Does the acquisition change anything for this integration? Not today. Both vendors say Dagster open source and Dagster+ continue with no migration required, and Data Workers uses each product's public API.
Can Data Workers re-execute a Dagster run from the failed step? Data Workers queues a new run of the named job. Re-execution from failure, partition backfills and terminating runs stay with your team in Dagster; for a backfill, Data Workers writes the plan and the owner launches it.
Sources
Sources for Dagster and Prefect capabilities and statuses, checked October 3, 2026: Dagster and Prefect ("Dagster is now part of Prefect", letter dated July 13, 2026), Prefect is acquiring Dagster (Nick Schrock, July 13, 2026), Prefect acquires Dagster Labs (Jeremiah Lowin, July 13, 2026), the Prefect homepage FAQ ("acquired Dagster Labs in July 2026"), the Prefect customer FAQ, the Dagster+ MCP server, Dagster skills and dagster-io/skills, Dagster+ AI proactive monitoring, Dagster+ Issues, the Dagster GraphQL API and Dagster+ GraphQL authentication, Dagster releases, Prefect agent resources, the Prefect MCP server guide, PrefectHQ/prefect-mcp-server and its `workflows` skill, Prefect AI log summaries, Prefect states, the Prefect REST API schema and Prefect releases. Product names and statuses change quickly; if we've got something wrong, tell us and we'll fix it.