Agent-built pipelines, tested before they ship
Pipeline requests wait in a queue while engineers rebuild the same boilerplate, and tests get written last, if at all. Data Workers builds pipelines from a plain-language request, tests them before deploy and ships them through review and approval.
Before and after
| Today | With Data Workers |
|---|---|
| A request queue measured in weeks | A request becomes a draft pipeline the same day |
| Boilerplate rebuilt by hand for every source | Pipelines written in your conventions from approved templates |
| Tests added last, or skipped | Tests written with the pipeline and run before deploy |
| Reviewers read a diff and guess the effect | Reviewers see what the change does to the data |
| Deploys by hand, one orchestrator at a time | Deploys through approval to the orchestrator you already run |
Every business team wants a new feed, and every data team has the same few engineers to build it. AI makes the first draft cheap, which moves the bottleneck to testing and review. Agent-built pipelines only help if the tests and the review come with them. For the single-agent view, see Inside the Pipeline Building Agent.
How it works
1. Describe the pipeline
You describe the pipeline in plain language, or hand over a ticket. The Pipeline Building Agent (generate_pipeline) drafts it from your templates (list_pipeline_templates) and your naming and modelling conventions. The spec is versioned in Git like any other code, so engineers keep working in the tools they use now, including Claude Code.
2. Test it before it ships
The Quality Monitoring Agent (create_quality_tests_for_pipeline) writes the tests that belong with the pipeline. The Pipeline Building Agent (validate_pipeline) runs the checks before anything is deployed. The Data Change Review Agent then compares what the change does to the data, so a reviewer sees the effect, not only the diff. On dbt projects this sits inside the pull request, next to what Data Workers already does with your dbt project.
3. Ship through review
A named approver signs off, and the Pipeline Building Agent (deploy_pipeline) deploys to the orchestrator you already run: Airflow, Dagster, Prefect or dbt Cloud. The Connectors Agent triggers the first run. Every step lands in the same audit trail, with the approver's name on the deploy.
The impact metrics we move
- •Request to reviewed pull request: from about 3 weeks to 2 days
- •Tests written per pipeline: from about 1 day to under 1 h
- •Change review prepared: from about 4 h to 15 min
Where teams start
- •dbt projects. New models drafted with tests, and the effect of each pull request shown before merge.
- •Airflow or Dagster estates. New DAGs generated from your own templates, deployed through approval.
- •Source-to-warehouse feeds. The recurring "land this table in Snowflake or BigQuery" request.
- •Analytics-engineering backlog. The oldest ten requests, built in the order the business ranks them.
- •Test coverage gaps. Existing pipelines that never got tests, covered one domain at a time.
What powers it
The Pipeline Building Agent drafts and validates, the Quality Monitoring Agent writes the tests, and the Data Change Review Agent shows the effect of a change on the data. The Data Context Wizard supplies what each table means and which source is trusted, so a generated pipeline reads the right copy. Autopilot Standards keep every new pipeline to your conventions, and the Autonomous Data-Conductor coordinates the agents and keeps what waits for approval in one view. The Spellbook Data Catalog records each new pipeline as it ships. Teams of data engineers see this in internal data products and data quality governance, and it pairs with schema changes that never break downstream.
The climb: from assisted to autonomous to agentic
Agents start by watching and suggesting: drafting pipelines and tests for people to review. On evidence, a domain can move to agents shipping routine pipelines inside approved templates, with a named approver on anything sensitive. L0 to L2 run today; L3 and L4 are where a domain can go as the track record builds. See our thesis.
The pilot
- •Scope. One domain and its ten oldest pipeline requests.
- •First week. Agents start by watching and suggesting: they draft pipelines and tests for your engineers to review, and nothing deploys.
- •By week two. Routine pipelines go through the full path, from draft to tested to approved. Anything sensitive stays with a named approver.
- •At day 30. We review the impact metrics against your baseline:
- •Request to reviewed pull request: from about 3 weeks to 2 days
- •Tests written per pipeline: from about 1 day to under 1 h
- •Change review prepared: from about 4 h to 15 min
- •Cost. Start with a pilot ($7,500 one-time, credited in full against the first year). Scale is from $1,000 a month and Enterprise from $3,000 a month, billed annually, with unlimited seats, no usage meter and no markup on model spend. See pricing.
FAQ
Does an agent deploy to production on its own? No. Deploys go through a named approver to your orchestrator, and every one is in the audit trail.
Which stacks does it work with? Deploys target Airflow, Dagster, Prefect and dbt Cloud; sources and warehouses connect through 50+ connectors. If you compare dbt with SQLMesh for agent work, this comparison covers it.
What if the agent writes a bad pipeline? The tests and the data comparison run before the deploy, and a reviewer sees the effect. The spec is in Git, so you review and revert it like any other code.
What stays with our engineers? Design decisions, sensitive pipelines and the approval. The agents take the boilerplate, the tests and the review prep.
Can our engineers keep using Claude Code or Cursor? Yes; the agents are reachable from the coding tools your team already uses.