Data platform teams: stop onboarding sources by ticket, start publishing products
Stop onboarding sources by ticket, start publishing products. Most of the week on data platform teams goes to upkeep: broken feeds, reruns, access queues and documentation. Data Workers hands that work to agents so the team ships the data products the business has been asking for.
Key takeaways
- •Who this is for. Teams that build the internal data platform itself: the governed platform that brings fleet, machine, cost and usage data together and publishes it as authoritative data products.
- •The problem today. A source-onboarding queue, contract breaks discovered by consumers and cost reviews nobody can act on safely.
- •What changes. Every producer deploy is checked against its data contract, so consumers stop finding breaks in production. Unused tables, oversized jobs and stale copies are cleaned up continuously after lineage proves they are safe, so spend stops growing faster than usage.
- •What stays with your people. Contract changes and new external consumers: these affect other teams and partners, so a platform owner approves them.
- •The result. A platform team that scales with its producers instead of becoming their bottleneck.
The PRD for agentic data platform and fleet data teams
This is the requirements document we would write with you for your team. Everything below expands on it.
| Who | Data platform PMs, infrastructure analytics, fleet data and platform data engineering teams |
| The job | Every contract enforced, every cost visible. |
| Environment | Snowflake, Databricks and BigQuery; Kafka; dbt; Airflow or Dagster; a catalog (Unity Catalog, DataHub, OpenMetadata); cost and usage telemetry. |
| Constraints | Contract changes and new external consumers always go to a named approver. Platform teams carry SOC 2 and internal audit obligations for changes. |
| Success is measured by | Partner consumer granted access: from about 2 weeks to 5 min; Contract break caught: from about 2 days to minutes; Finding a data product: from about 4 hours to 30 s; catalog upkeep closed autonomously 90% of the time. |
| Owned by | A Data Platform PM, with domain owners as named approvers. |
| Pilot scope | A cost mission on one domain. |
What going fully agentic and autonomous means for data platform and fleet data teams
- •Put your data pipeline on autopilot. Every producer deploy is checked against its data contract, so consumers stop finding breaks in production.
- •Run the data back office without the busywork. The work that fills the week today moves to agents: a source-onboarding queue, contract breaks discovered by consumers and cost reviews nobody can act on safely.
- •Scale data without scaling headcount. Unused tables, oversized jobs and stale copies are cleaned up continuously after lineage proves they are safe and new sources are onboarded to staging with profiling, a contract and documentation, so the data you manage grows without the team having to.
- •Get the plumbing off your plate. A platform team that scales with its producers instead of becoming their bottleneck.
Who this is for
Teams that build the internal data platform itself: the governed platform that brings fleet, machine, cost and usage data together and publishes it as authoritative data products. You find them at cloud providers, GPU and infrastructure companies, SaaS platforms and any large engineering org. Titles include Senior Data Platform and Analytics PM, Platform Data Engineering Lead and Head of Data Platform.
The product you ship
- •Governed data products that answer cross-fleet questions with one query.
- •Data contracts with every producing team, versioned and enforced.
- •Standard identifiers and schemas across clusters, regions and services.
- •Platform metrics for capacity, utilization, reliability and cost.
- •External access for partners and customers on the same platform.
Where the week goes
- •Every producing team ships its own schema. Identifiers do not join across sources.
- •Contracts are negotiated and then ignored. Producers change fields and consumers find out in production.
- •Source onboarding is a queue. Each new source takes weeks of engineering.
- •Security and compliance reviews gate every new consumer.
- •Cost grows with every source.
Add it up and upkeep is most of the team's week. Here is how a typical week splits today, and once agents own the back office:
Time spent on new data products moves from 18% of the week to 52%. Most of the rest becomes review: people steer and approve agent work instead of doing it by hand.
Then and now
| Work | Today | With Data Workers |
|---|---|---|
| Producer schema changes | Consumers find out in production | Contract checked on every deploy |
| New source onboarding | Weeks per source | A mission run in days |
| Identifiers across products | Do not join | Standardized and maintained |
| Cost per data product | Unknown | On the dashboard |
| Cleanups | Abandoned for fear of breaking things | Run after lineage proves safety |
| External partner access | A separate project | The same policies, extended |
These are the impact metrics we move for data platform and fleet data teams:
- •Partner consumer granted access: from about 2 weeks to 5 min
- •Contract break caught: from about 2 days to minutes
- •Finding a data product: from about 4 hours to 30 s
Across the routine work in this domain:
The biggest change is Partner consumer granted access: from about 2 weeks to 5 min. Work that waited in queues finishes in minutes or hours, because agents detect, fix and verify in one pass.
Use cases, in depth
A node identifier rename
Today. Fleet dashboards break and the platform team finds out from consumers.
With Data Workers. Schema Evolution catches the contract break on deploy, Pipeline Building updates the mapping, Quality Monitoring confirms join rates and the producer is notified with the contract version.
What changes. Consumers never see the break.
Onboarding 20 telemetry sources
Today. Each source waits in the platform team's queue.
With Data Workers. Autopilot runs onboarding as a mission: connect, profile, contract, model and document each source.
What changes. The platform grows at the pace of its producers.
A cost mission across 100 pipelines
Today. Spend climbs every quarter and nobody can prove what is safe to touch.
With Data Workers. The swarm finds waste, proves dependencies, changes in waves and verifies every SLA.
What changes. Spend falls without missing an SLA.
Agents working together: A cost mission across 100 pipelines
Every data platform has the same cost story. Spend climbs every quarter, everyone knows there is waste, and nobody can say what is safe to touch. Cleanups get scoped, staffed and abandoned because proving nothing breaks takes longer than the savings are worth. This is a job agents should own.
The mission
A head of data platform sets one outcome in Autopilot:
Reduce warehouse and compute spend on the marketing and product analytics domains by 25% this quarter without missing any freshness SLA.
Autopilot turns that into a mission across roughly 100 pipelines and shows two numbers before anything starts: the estimated token cost of running the mission and the estimated monthly saving if it succeeds. The team decides it is worth it.
Week 1 — find the waste
Cost Savings & Data Cleanup agent inventories everything in scope across Snowflake, Databricks and BigQuery: tables nobody has read in 90 days, duplicated sandboxes, full refreshes that could be incremental, oversized clusters, and schedules that run hourly for dashboards opened weekly. It ranks them by saving.
Week 1 — prove what depends on it
Before anything is changed, Data Context Wizard runs every candidate through lineage: which models, dashboards, ML features, reverse-ETL syncs and agents read it. Anything with a live consumer is excluded or marked for a careful change. This is the step manual cleanups skip, and the reason they fail.
Weeks 2 to 4 — change in waves
- •Cost Savings & Data Cleanup drops abandoned sandboxes and unread tables at L4, with a retention snapshot for rollback.
- •Pipeline Building agent rewrites the worst full-refresh models as incremental and adds the tests to prove row-level parity with the old output.
- •Schedules are matched to consumption: hourly jobs feeding a weekly dashboard move to daily.
- •Cluster and warehouse sizes are right-sized against the actual query profile.
Every change ships as a wave, small enough to roll back, with a signed receipt.
Every week — verify nothing broke
Quality Monitoring agent compares outputs before and after each change. Autonomous Data-Conductor watches every freshness SLA in scope and every downstream number that matters. One rewritten model runs 40 minutes later than before because it now waits on an upstream table; Conductor catches it on the first run, rolls the schedule change back and tries a different slot.
The result the team sees
The Autopilot dashboard shows, per domain:
- •Spend before, spend now and the projected quarterly saving.
- •Every change made, by which agent, at which autonomy level.
- •Token cost of the mission against the saving it produced.
- •SLAs met through the whole campaign.
The cost work also becomes a standing routine. New waste is caught the week it appears instead of the quarter after.
Why this is a swarm job
No single agent could do it. Finding waste is one skill, proving dependencies is another, rewriting models safely is a third, and verifying outcomes is a fourth. Data Workers coordinates them under one outcome, with autonomy you set, across every cloud you run.
Minute by minute
This is an illustrative example of a cost mission across 100 pipelines on Snowflake and Databricks.
| Time | Who | What happens |
|---|---|---|
| Week 1 Mon | Autopilot | Mission stated: cut spend 25% without missing an SLA; token cost and saving shown |
| Week 1 | Cost Savings & Data Cleanup agent | get_snowflake_usage and get_cost_enrichment rank waste |
| Week 1 | Data Context & Catalog agent | detect_dead_assets plus blast_radius_analysis prove consumers |
| Week 2 | Quality Monitoring agent | Unread tables dropped at L4 with retention snapshots |
| Week 2-4 | Pipeline Building agent | Full refreshes rewritten as incremental with parity tests |
| Week 3 | Autonomous Data-Conductor | One schedule change misses an SLA; rolled back automatically |
| Week 4 | Autopilot | Dashboard shows spend, savings and SLAs met |
The agents on the job
| Agent | What it does for data platform teams | Tools it calls |
|---|---|---|
| Connectors and Data Ingestion | Onboard new sources and profile them against the platform's standards | get_connector_status, trigger_airflow_dag, get_dbt_run_history |
| Schema Evolution | Enforces data contracts and versions every producer change | detect_schema_change, validate_schema_compatibility, generate_migration, rollback_migration |
| Pipeline Building | Builds and standardizes data products and identifier mappings | generate_pipeline, validate_pipeline, sandbox_test, deploy_pipeline |
| Quality Monitoring | Contract, freshness and join-rate checks on every data product | set_sla, run_quality_check, get_anomalies, create_quality_tests_for_pipeline |
| Data Access & Governance | Applies the platform's security model to internal and partner consumers | provision_access, check_policy, request_governance_review, generate_audit_report |
| Cost Savings & Data Cleanup | Keeps storage and compute per data product in check | get_snowflake_usage, get_cost_enrichment, detect_dead_assets |
Context Wizard holds the standard identifiers and definitions every product joins on. Conductor watches every contract and SLA and handles breaks. Spellbook is the platform's catalog, with every product's owner, contract version, SLA and consumers on one page. Autopilot shows autonomy and cost per data product.
What goes on autopilot
Once a domain has earned it, this work runs on its own, and each item removes a problem your team deals with today:
- •Every producer deploy is checked against its data contract, so consumers stop finding breaks in production.
- •Unused tables, oversized jobs and stale copies are cleaned up continuously after lineage proves they are safe, so spend stops growing faster than usage.
- •New sources are onboarded to staging with profiling, a contract and documentation, so the queue disappears.
Where your people stay in charge
Agents never get blanket access. Each domain has its own autonomy level, set by you, and it only rises when the receipts show the agents are reliable. Everything starts at L1, observing only; reversible upkeep such as reruns, staging fixes, documentation and cleanup is usually first to run on its own.
- •Contract changes and new external consumers always goes to a named approver: these affect other teams and partners, so a platform owner approves them.
- •Changes to the security model: these stay with people.
Once a deployment has matured, here is how that ladder plays out across task classes:
Catalog upkeep closes autonomously 90% of the time. For Contract changes and new external consumers, autonomous closure is 0% by design: every change is proposed with a full diff and approved by a named person.
What happens when an agent is wrong
- •At L2 propose, nothing changes until a named person approves. The proposal shows the full diff, the blast radius and how to roll it back. Unanswered requests expire and escalate.
- •At L3 act, reversibly, every change keeps a rollback point, and Conductor verifies the downstream number after the change. If verification fails, the change is reversed and the domain drops a level.
- •No agent approves or promotes its own work. Autonomy only grows on evidence, per repair class, and repeated errors shrink scope automatically.
- •One org-wide stop halts all autonomous dispatch, and every action lands in a hash-chained audit trail you can export with generate_audit_report.
How it fits your stack
Data Workers connects to what you already run: Snowflake, Databricks and BigQuery; Kafka; dbt; Airflow or Dagster; a catalog (Unity Catalog, DataHub, OpenMetadata); cost and usage telemetry. Each agent is an MCP server, so the same agents work inside Claude Code, Cursor or Codex and in Slack or Teams through Spellbook. Nothing migrates.
- •Runs in your infrastructure with your credentials and your model key. The hosted Autonomous Data-Conductor sees workflow metadata only, never rows.
- •Uses the access you already govern. Agents act under scoped service identities and your existing roles; every grant goes through provision_access and is logged.
- •Proposes changes as diffs in your dbt project, orchestrator and warehouse, with blast radius, tests and a rollback point, and applies them only at the autonomy level you set for that domain.
- •Open-source core. The Apache-2.0 community edition ships 11 agents and 160+ MCP tools, so your engineers can read exactly what the agents call.
Governance and regulation
Platform teams carry SOC 2 and internal audit obligations for changes. Data Workers records every change with a receipt and exports the trail with generate_audit_report.
Data Workers doesn't make you compliant; your controls, people and assessors do. It supplies controls and evidence wherever agents touch data. Not legal advice.
The alternatives data platform and fleet data teams weigh
Build it yourselves. A strong team can wire a coding agent to MCP servers for the warehouse and dbt. The work is everything around it: routing approvals to the right owner, blast radius, rollback, receipts, and current context about every other system.
Observability, catalogs and point tools. Observability alerts, catalogs document, and quality tools test. Data Workers takes their signals, diagnoses, proposes the fix, verifies it and leaves the receipt.
Platform-native agents. Snowflake made its managed MCP server and Cortex Agents generally available on Nov 4, 2025, and Databricks made Genie One generally available on Jun 16, 2026. They are excellent inside their platform; this work crosses the source systems, orchestrator, warehouse and BI, and Data Workers works across all of them.
In this space specifically. Cost tools report spend; catalogs list assets. Data Workers proves dependencies, makes the change and verifies SLAs afterwards.
The case for your CTO
What this replaces. A source-onboarding queue, contract breaks discovered by consumers and cost reviews nobody can act on safely.
What your team gets back. A platform team that scales with its producers instead of becoming their bottleneck.
Integration effort. Connecting is configuration, not a migration. The agents attach over MCP to the warehouse, orchestrator, dbt project and source systems you already run, under the roles you already govern, inside your own infrastructure. Your engineers can read every tool the agents call in the open-source core.
Risk and control. Autonomy is set per domain. Everything starts at L1, observing only; reversible work moves up only when the receipts justify it; contract changes and new external consumers stay at L2 with a named approver; and one org-wide stop halts all autonomous work.
Why now. Data platform spend keeps growing faster than usage.
The first win. A cost mission on one domain. The pilot below lays out the scope, what you see each week and the cost.
The climb: from assisted to autonomous to agentic
Our thesis is that the data platform climbs the way analytics did, in stages, and that the hard part of data work was never knowing what to do but doing it. For data platform and fleet data teams, the climb looks like this:
- •Assisted. Agents work alongside your team. They read the whole stack, show what needs doing and propose every fix with a receipt; contract changes and new external consumers always go to a named approver.
- •Autonomous, under your governance. Routine work runs on its own: every producer deploy is checked against its data contract, unused tables, oversized jobs and stale copies are cleaned up continuously after lineage proves they are safe, and new sources are onboarded to staging with profiling, a contract and documentation. Your team governs the agents instead of doing the upkeep.
- •The agentic enterprise. A platform team that scales with its producers instead of becoming their bottleneck. People set direction, and the platform carries it out.
You choose the altitude, every step is visible and reversible, and the pilot below is the first rung.
The pilot: how data platform and fleet data teams start
The pilot is how your team proves the value on its own data before committing.
- •What we connect. Snowflake, Databricks and BigQuery; Kafka; dbt; Airflow or Dagster; a catalog (Unity Catalog, DataHub, OpenMetadata); cost and usage telemetry, over MCP, inside your infrastructure. Nothing migrates.
- •Scope. A cost mission on one domain.
- •First week. Agents run at L1 across the scoped domain. They map every pipeline, definition and lineage path and show the fixes they would have made, with receipts, so your team sees the value before anything changes.
- •By week two. Every producer deploy is checked against its data contract and unused tables, oversized jobs and stale copies are cleaned up continuously after lineage proves they are safe start running on reversible work. Contract changes and new external consumers stay at L2 with a named approver.
- •At day 30. We review the impact metrics against your baseline:
- •Partner consumer granted access: from about 2 weeks to 5 min
- •Contract break caught: from about 2 days to minutes
- •Finding a data product: from about 4 hours to 30 s
- •the share of catalog upkeep closed autonomously, targeting 90% as the deployment matures
- •What you decide. Whether to expand to the next domain. The pilot fee is credited in full against the first year.
- •Cost. Start with a pilot ($7,500 one-time, credited in full against the first year). Scale is from $1,000 a month and Enterprise from $3,000 a month, billed annually, with unlimited seats, no usage meter and no markup on model spend. See pricing.
The pilot is the start of a longer curve:
By month 12, more than 50% of agent actions run fully autonomously and manual work falls below 10%, while contract changes stay with a named approver.
FAQ
Will it delete data? Only at the level you set, with snapshots for rollback.
Does it work across clouds? Yes: Snowflake, Databricks and BigQuery.
Can we see what it saved? Spend before and after is on the Autopilot dashboard.
Where do the agents run? In your infrastructure, with your credentials and model key. The hosted Conductor sees workflow metadata only.
Do we have to migrate anything? No. Your warehouse, dbt project, orchestrator, catalog and BI stay where they are.
Sources
- •NVIDIA, Senior Data Platform and Analytics PM, DGX Cloud: https://www.builtinseattle.com/job/senior-data-platform-and-analytics-product-manager-dgx-cloud/11431450
- •Snowflake, Sr PM Data Observability: https://zapply.jobs/jobs/c6879916-c1a7-46ba-9143-bae2b08916ce/
- •Data Workers pricing: https://dataworkers.io/pricing/
- •Data Workers open-source repository (tool registrations): https://github.com/DataWorkersProject/dataworkers-claw-community
- •Snowflake, managed MCP server and Cortex Agents GA release note (Nov 4, 2025): https://docs.snowflake.com/en/release-notes/2025/other/2025-11-04-cortex-agents-mcp
- •Databricks, Genie One launch (Jun 16, 2026): https://www.databricks.com/company/newsroom/press-releases/databricks-launches-genie-one-all-new-agentic-coworker-every-team