For Airflow
Data Workers runs the detect, diagnose, fix, review and verify loop on an Airflow or Dagster deployment when nobody is on call. A task that failed its last retry at 3am is diagnosed against the data it was moving, a fix is proposed, a named human approves anything irreversible, and every applied change carries a receipt and a rollback.
For a data team whose orchestrator is Airflow, Dagster or both: DAGs and assets, retries and sensors, backfills, and an on-call rota that gets paged for things a person cannot fix at 3am anyway.
Last updated: September 10, 2026 · Dhanush Shetty, founder, Data Workers
None of these is a Airflow defect. They are the failures a busy estate produces, and the hours between one of them starting and somebody verifying a fix are what this platform is for.
A task exhausted its retries and the retry was never going to work
Three retries against an upstream that changed shape is three identical failures and forty minutes of clock. The orchestrator is doing exactly what it was told. Nobody told it to read the error, which is the only thing that would have helped.
A DAG succeeded and the data is wrong
This is the failure orchestration cannot see by construction. Every task returned zero, the run is green, and the table has a third of the rows it should. The orchestrator reports on execution; nothing in it reports on the thing that was executed.
An SLA miss fired and nobody can say what it affected
The alert names a task. Answering the actual question, which is which dashboards and which downstream models are now stale, means someone who knows the estate reading the DAG and then reading the warehouse. That person is asleep.
A backfill is running and nobody is sure what it is about to overwrite
Backfills are the highest-blast-radius routine operation in data engineering and the one most often run from memory. The partitions it will touch and the consumers of those partitions are knowable, and they are almost never known before the command is entered.
A zombie task holds a pool slot and everything behind it queues
The worker died without telling the scheduler. The task shows running for hours. Everything competing for that pool is late, and the first symptom is a freshness complaint from a team three hops downstream.
Alerts are not fixes. Detection tells you the first of those sentences. The rest is the work.
Runs the loop
Autonomous Data-Conductor
The Autonomous Data-Conductor sits beside the orchestrator rather than replacing it. Airflow decides what runs and when, and it is good at that. The Conductor takes over at the point Airflow stops: a task that has failed its last retry, an SLA miss, a run that succeeded while its output failed a quality check. It then diagnoses against the data rather than the log alone, proposes a fix, and runs it past whatever gate you set. You set the autonomy dial per class: read-only, propose-and-wait, or apply-then-report for reversible work such as clearing a task or triggering a targeted rerun. A backfill is never reversible and always stops for a named human.
Does the work and writes
Data-Agents Swarm, 20 specialised agents
The Data-Agents Swarm is 20 specialised agents. Around an orchestrator the ones that matter first are incidents, which does graph-based root cause across the task and the tables it touched; schema, which catches the upstream change that made the retry pointless; quality, which is what turns a green run with wrong data into a detected event at all; and observability, which keeps a hash-chained record of what the agents did. The open-source core reads and recommends. Writing is the paid tier.
What the agents reason on
Data Context Wizard
The Data Context Wizard is the graph the agents reason on, and against an orchestrator its job is specific: the DAG knows the order of operations, and the graph knows what those operations mean. It holds column-level lineage through the tables a task writes, metric definitions, ownership that reflects who actually answers, and past corrections. That is what lets an agent answer which dashboards an SLA miss affected, which is a question the orchestrator cannot answer at all because it has never looked inside the tables.
Where humans approve and audit
Spellbook Data Catalog, in preview
Spellbook Data Catalog, in preview, is where a human approves an agent's proposed change and where the audit trail lives afterwards. For an orchestration team the approval that matters most is the backfill: what partitions, what consumers, who said yes, and how to undo it.
What each module has to do to qualify as this kind of product, stated generically so you can use it on any vendor: the autonomous agentic data platform.
The first three steps need no form, no account and no key we issue. The open-source core is Apache-2.0: 11 agents and 160+ MCP tools that read, analyse and recommend.
The full list of who we are not for, including the cases where the honest answer is to buy something else: should you use Data Workers.
Does this replace Airflow?
No, and it would be a bad trade if it did. Airflow decides what runs and when, and that is a solved problem you have already solved. This starts where Airflow stops: the task failed its last retry, or the run went green while the data went wrong, and now somebody has to work out why and what to do.
We are on Dagster, not Airflow. Does that change anything?
The argument is the same and some of the overlap is larger. Dagster's asset model and asset checks already express more about what a job produces than a DAG does, so parts of the quality agent duplicate something you have. What does not change is that neither tool reads your warehouse to diagnose a root cause or knows which dashboards a miss affected.
Does it need access to our Airflow instance?
The read-only core does not: it works against the warehouse and the metadata, which is where the diagnosis actually happens. Acting on the orchestrator, such as clearing a task or triggering a rerun, needs the orchestrator's API and is part of the paid write path, gated the same way as every other write.
Can it change production without us?
Not for anything irreversible, and a backfill counts as irreversible whatever your dial says. The write path is propose first, dry run where one exists, and a named human approves. Every applied change carries a receipt: the diff, the approver, the blast radius across downstream tables and dashboards, and the rollback path.
Two next steps
Run the read-only agents on Airflow yourself, free: clone the open-source core.
Or bring one real incident to a 45-minute session and watch the approval gate stop the agent: book a time. Pilot Program $7,500 one-time, then Scale from $1,000 per month, no usage meter, as published on 10 September 2026. Pricing.