Data Workers for data platform engineers
How Data Workers changes a data platform engineer's week: policy-checked access requests, pipeline incidents with a root cause attached, cost findings, and migrations with parity and rollback SQL, running in your environment under your approvals.
Data Workers takes the access-request queue, the 2am pipeline triage, the cost hunt and the migration legwork off a data platform engineer's plate, and hands each one back as a proposal with a diff, a blast radius and a rollback path. The platform engineer still owns the platform: where the agents run, what their credentials can reach, the autonomy level of every domain and every approval that matters.
Key takeaways
- •Your environment, your credentials. The agents run in your infrastructure on every tier and hold your warehouse credentials and model key. Your data stays in your systems; the hosted Autonomous Data-Conductor sees workflow metadata only.
- •Access requests arrive ready to approve. Policy-checked, least-privilege grants with an expiry date, routed to the data owner, with nothing granted when policy says review. On Unity Catalog, the approved grant is applied for you.
- •On-call starts with a diagnosis. A failed DAG run comes with its root cause, its blast radius and a proposed fix.
- •Migrations move in gated waves. Parity checks planned and tracked per wave, a completion gate held for your sign-off, every reader mapped, and migration SQL generated with rollback SQL for you to apply.
- •You set the dial. Autonomy runs from L0 manual to L4 autonomous, per domain, and Data Workers ships observe-only.
The week today, and the same week with Data Workers
The planned work is Terraform for warehouses and roles, CI/CD for data, orchestrator upgrades and a Glue to Unity Catalog move. The unplanned work eats it. Access tickets in Jira or ServiceNow become hand-written GRANT statements. PagerDuty fires at 02:10 for a failed Airflow DAG, and the first hour goes on finding the upstream change. Finance asks why the Snowflake bill went up. The migration tracker is a spreadsheet, and nobody is sure which dashboards still read the old tables.
In the dbt Labs 2026 State of Analytics Engineering report, 57% of respondents reported increased warehouse and compute spend against 36% reporting increased team budgets, and only 24% prioritise AI-assisted pipeline management. The FinOps Foundation's State of FinOps 2026 found 78% of FinOps teams report to the CTO or CIO, so cost questions land on the platform team. With Data Workers, the legwork moves to the agents and the decisions stay with you.
What you still own and decide. The deployment and the scope of the credentials the agents hold. The autonomy level per domain: L0 manual, L1 observe, L2 propose, L3 act reversibly, L4 autonomous. Who approves what: approvals go to a named person, an unanswered request expires and escalates, never auto-grants, and no agent can promote its own work. The org-wide stop halts all autonomous dispatch, and the admin kill switch needs two people and revokes the tenant's keys, tokens and sessions. See how approvals work and who owns the agents.
What you get back. A receipt on every change (the diff, the approver, the blast radius, the rollback path) in a hash-chained audit log; verify_global_hash_chain checks the agents' SHA-256 chains end to end.
Where it runs, and what the agents can touch
The agents run in your infrastructure on every tier, and the context graph, receipts and audit log are written there. Enterprise adds a dedicated VPC, your own cloud or on-premise, and an air-gapped build on local models with sovereign mode, which blocks external model calls. See Data Workers in your VPC or air-gapped and where does our data go?
What the agents change is narrow by design:
| Area | What Data Workers does | What stays with you |
|---|---|---|
| Unity Catalog | Applies approved grants and revocations through its permissions API | The grant model and who approves |
| PostgreSQL | Writes the GRANT statement for an approved request | Running it |
| Snowflake, BigQuery, Lake Formation | Proposes grants and masking policies | Applying them through your change process |
| Schemas | Generates migrations with rollback SQL | Running them; nothing executes them for you |
| Orchestrators | Queues a rerun through Airflow, Dagster, Prefect or ADF | The DAGs and schedules |
| Terraform | Cost guardrails as specs you render into code | Your repo, your pipeline, your apply |
Every grant is policy-checked first, carries an expiry date and lands in the audit log, and every action passes the safety model of blast-radius scoping, dry run and rollback.
The tools you already run, and how Data Workers fits them
Snowflake, Databricks with Unity Catalog, BigQuery, PostgreSQL, Kafka, Airflow, Dagster, Prefect, ADF, dbt, Datadog, PagerDuty, ServiceNow, Okta and Entra ID connect natively, among 50+ connectors. Iceberg REST catalogs, Glue and Lake Formation, Fivetran, Airbyte and other ingestion tools connect over their APIs or MCP servers today; Data Workers watches the schemas they land and leaves the syncs to them. See the integrations page and the guides for Snowflake, Databricks and AWS.
You work with Data Workers in three places. In your coding agent, the agents are MCP servers, so Claude Code, Cursor or Codex call diagnose_incident or blast_radius_analysis from the open session. In pull request review, Data Workers adds a blast radius and an impact diff and blocks risky changes. In Spellbook Data Catalog (in preview), approvers review, approve, roll back and audit.
Setup follows the client setup docs: clone the open-source core, install it, and add one start-agent.sh entry per agent to your client's MCP config.
# Example: register the agents a platform engineer uses most in Claude Code
git clone https://github.com/DataWorkersProject/dataworkers-claw-community.git
cd dataworkers-claw-community && npm install --ignore-optional
claude mcp add dw-incidents -- "$(pwd)/start-agent.sh" dw-incidents
claude mcp add dw-governance -- "$(pwd)/start-agent.sh" dw-governance
claude mcp add dw-schema -- "$(pwd)/start-agent.sh" dw-schemaThe core runs on a sample estate; a Data Workers deployment points the same agents at yours, with approvals and receipts. List the tools with your client's own command, such as /mcp.
Access requests. check_policy evaluates the request and provision_access records a least-privilege grant with a 90-day expiry by default; when the verdict is review it grants nothing, and request_governance_review opens a trackable review for the data owner. On Unity Catalog, the approved grant and its later revocation go through the permissions API.
Cost. The cost agent reads Snowflake metering, attributes credits to the query and the dbt model, project and run behind them through query tags, reads BigQuery spend from the Jobs API and AWS cost by service from Cost Explorer, and drafts warehouse settings for you to apply. For a new Snowflake warehouse it recommends an extra-small default, a 60-second auto-suspend and a resource monitor with a credit quota, as a spec you render into Terraform.
Migrations. The migration agent translates SQL, plans each wave with its parity checks, proposes the comparison queries your team runs in its warehouses, tracks the results and holds the completion gate until validation and parity are signed off by the owner. It catches a masking tag dropped when a column is renamed and proposes the repair. generate_migration and rollback_migration write the forward and rollback SQL for you to apply. Glue to Unity Catalog with parity checks walks a full move.
Incidents. diagnose_incident, get_root_cause and get_incident_history explain the failure; remediate proposes the fix at the level you set; tickets land in ServiceNow or Jira Service Management. See the PagerDuty and Datadog pages and the data reliability engineering guide.
The metrics you are judged on
Platform engineers are measured on SLAs, time to resolve, cost per workload, time to provision access, audit findings and migration milestones. How Data Workers moves them, as design targets rather than measured results:
- •Time to resolve falls because the diagnosis and a fix are waiting when the page arrives.
- •Time to provision access falls from a ticket queue to one approval.
- •Cost per workload falls as each fix reaches the owner of the model behind the credits; the design target is 25 to 40% lower warehouse spend.
- •Migration milestones come in earlier with parity checks planned and tracked per wave; the design target is 4 to 8 weeks for a migration teams usually plan at 6 to 12 months.
- •Audit findings shrink because every grant has an owner and an expiry date, and every change a receipt.
How to measure AI data agents sets up the baseline, and the ROI calculator models your estate.
A worked example: a Unity Catalog wave and the reader it missed
This is an illustration, not a customer case. A platform team is moving finance tables from the AWS Glue Data Catalog to Unity Catalog in waves. Airflow schedules the jobs, dbt models the tables, Tableau reads them, and PagerDuty pages the platform on-call. The migration domain runs at L2 propose.
| Time | System | What happened |
|---|---|---|
| 14:00 | Databricks | Wave 3: 42 finance tables are synced from Glue to Unity Catalog |
| 14:20 | Data Workers, Databricks | The team runs the parity queries Data Workers proposed for all 42: row counts, schemas and aggregates match. Data Workers logs the results against the wave and flags one dropped masking tag |
| 14:35 | Data Workers | trace_cross_platform_lineage maps the readers: 3 Airflow DAGs, 6 dbt sources and 2 Tableau workbooks still use the Glue names |
| 14:50 | Data Workers | Proposes repoint diffs and the masking tag repair, each with its rollback path |
| 15:10 | Spellbook | The platform engineer approves the repoints |
| 15:25 | Spellbook | The governance lead approves the tag repair, and the owner applies it |
| 15:40 | Data Workers, Airflow | Reruns are queued through Airflow; the team's parity rerun matches and Data Workers logs it at the wave's completion gate |
| 02:10 | Airflow | A notebook task with a hard-coded Glue path fails |
| 02:12 | Data Workers | get_root_cause traces it to the path; a one-line fix is proposed with rollback |
| 02:15 | PagerDuty | The on-call engineer is paged with the diagnosis and the diff |
| 02:22 | Spellbook | The on-call engineer approves; the rerun is queued and the DAG goes green |
The 02:10 page was closed by 02:22, and the receipt holds the parity results, the diffs, the approvers and the rollback path for every step.
What changes for the platform team
The platform team stops being the queue in front of the platform and spends its time on design, upgrades and guardrails. Will Data Workers replace my data team? has the short answer: it takes the toil; engineers own the design and the approvals. See also Data Workers for data engineers, Data Workers for FinOps leads and the platform engineering for data guide.
How to bring it to your team
Install the agents in your coding agent against a non-production estate and run them at L1 observe for two weeks: they log what each action would have needed, and you review the records and receipts against the bar your team sets. Then move one domain to L2 propose. Access requests and pipeline incidents are the usual first two, because the before and after are easy to count.
Bring your lead the deployment answer, the approval model and the observe-period numbers. Autonomy levels L0 to L4, how to get your team to trust AI agents, ROI of agentic data operations and build it ourselves with Claude Code and MCP servers answer the next questions. The path to your real estate is a pilot: pricing is on /pricing/, and the pilot is credited in full against the first year.
FAQ
Do the agents need admin credentials on our warehouse? No. Each agent holds its own scoped credentials that you issue, and acts only inside that scope and its autonomy level.
Will Data Workers apply Terraform or run DDL in production? No. Migrations come with rollback SQL for you to apply, Snowflake and BigQuery grants and masking are proposed for the owner, and cost guardrails arrive as Terraform-ready specs. Unity Catalog grants are applied after approval, and reruns are queued through your orchestrator.
How do we stop it if something goes wrong? Use the org-wide stop or the two-person kill switch described above. Every change has a rollback path in its receipt; how to roll back an AI agent change shows the steps.
Does it work with our identity provider? Yes. Okta and Entra ID connect natively and are read for users and group membership, and remote endpoints verify OAuth tokens from your provider with JWKS.
Which model does it use? Yours. You bring the model key, with no markup on model spend and no usage meter, and air-gapped builds run on local models. See which model Data Workers uses.
Sources
- •dbt Labs, 2026 State of Analytics Engineering Report ("57% report increased warehouse and compute spend, compared to 36% reporting increased team budgets"; 72% prioritise AI-assisted coding, 24% AI-assisted pipeline management): https://www.getdbt.com/resources/state-of-analytics-engineering-2026 (checked Oct 2, 2026)
- •FinOps Foundation, State of FinOps 2026 ("78% of teams now report to the CTO or CIO"): https://data.finops.org/ (checked Oct 2, 2026)
- •Databricks, Hive metastore (legacy status, disabling direct access), updated Sep 11, 2026: https://docs.databricks.com/aws/en/data-governance/unity-catalog/hive-metastore (checked Oct 2, 2026)
- •Snowflake, Working with resource monitors: https://docs.snowflake.com/en/user-guide/resource-monitors (checked Oct 2, 2026)
- •Data Workers public repository tools (
provision_access,check_policy,request_governance_review,diagnose_incident,get_root_cause,remediate,get_incident_history,generate_migration,rollback_migration,trace_cross_platform_lineage,blast_radius_analysis,verify_global_hash_chain): https://github.com/DataWorkersProject/dataworkers-claw-community (checked Oct 2, 2026) - •Data Workers client setup: https://dataworkers.io/opensource-docs/client-setup/ (checked Oct 2, 2026)
- •Data Workers product repository,
data-workers-agent-swarmmain: agents/dw-governance (policy-gated, approval-gated grants; Unity Catalog grant and revoke through the permissions API; PostgreSQL GRANT statement), agents/dw-cost (Snowflake metering and per-query attribution with dbt query tags, provision-time guardrails), agents/dw-migration (translation, parallel comparison, parity bisect, completion gate, masking tag repair), agents/dw-review (pull request review), core/enterprise (approvals, promotion guard, audit log) (checked Oct 2, 2026) - •Data Workers product pages and ROI calculator: https://dataworkers.io/product/autonomous-data-conductor/ , https://dataworkers.io/product/spellbook-data-catalog/ , https://dataworkers.io/product/data-agents-swarm/ , https://dataworkers.io/roi-calculator/ (checked Oct 2, 2026)
- •Data Workers security and pricing: https://dataworkers.io/security/ , https://dataworkers.io/pricing/ (checked Oct 2, 2026)