ML platform data teams: ship models on data that never silently drifts

Key takeaways

• •Who this is for. The team that owns the data under every production model: feature pipelines, the feature store, training and evaluation datasets, model registry inputs, drift monitoring and the data contracts between data engineering and ML.
• •The problem today. A source team renames a column, the pipeline stays green, the feature quietly goes to zero and the model degrades for weeks before anyone connects the two. Training sets that cannot be rebuilt as of the right date. Drift alerts nobody can trace to a cause.
• •What changes. Upstream schema changes are mapped into feature pipelines the day they ship, with the affected features and models listed before anything runs. Feature freshness and skew are checked every morning, and training sets are rebuilt point in time with tests the team defines.
• •What stays with your people. Promoting a model to production, approving retraining, and access to labels and sensitive features. The ML platform lead approves each with a full diff and a rollback point.
• •The result. An ML platform team that ships models and evaluation instead of chasing a broken feature, and a model fleet whose upstream data never changes without someone knowing.

The PRD for agentic AI and ML platform data teams

This is the requirements document we would write with you for your team. Everything below expands on it.

What going fully agentic and autonomous means for AI and ML platform data teams

• •Put your feature pipelines on autopilot. Upstream schema changes are mapped into the features that read them the day they ship, so a Friday rename in a source system never becomes a Monday model regression.
• •Run the data back office without the busywork. The work that fills the week today moves to agents: tracing a degraded model to its cause, re-running late features, rebuilding training sets and writing the lineage evidence for model reviews.
• •Scale models without scaling headcount. Freshness, skew and null-rate checks cover every feature view every morning, and lineage from source to feature to model is kept current as the fleet grows.
• •Get the plumbing off your plate. An ML platform team that ships models, evaluation and new capabilities instead of finding out which table a drift alert came from.

Who this is for

The team that owns the data under production machine learning. At a software or internet company it is the ML platform or MLOps team inside a central AI organization, serving applied scientists and product teams that train recommendation, ranking, fraud, forecasting and, increasingly, retrieval and agent workloads. At a bank or an insurer it is the model engineering group that must show where every score came from. At a manufacturer, a retailer or a health system it is the small platform team behind a dozen models that matter to the business.

The titles: Head of ML Platform, Director of MLOps, Staff ML Platform Engineer, Feature Store Product Manager, ML Data Engineering Lead and Applied Science Manager. They sit between data engineering, which owns the sources, and the model teams, which own the metrics, and they are the first call when a model gets worse and nobody knows why. The data teams upstream of them are covered in the data platform playbooks; the group overview is on the AI and ML data teams page.

The product you ship

• •Features: fresh, versioned, documented feature definitions with an owner, a source and an expected range, served consistently to training and to inference. For the basics, see what a feature store is.
• •Training and evaluation datasets: point-in-time-correct, reproducible and versioned, so a result from March can be rerun in October.
• •Model registry inputs: the exact data, code and feature versions behind every registered model.
• •Drift and skew monitoring: a clear signal for whether a model got worse because the world changed, the data changed or the pipeline broke.
• •The data contracts between data engineering and ML: agreed columns, meanings, units and freshness that source teams can see before they change something.
• •Lineage from source to feature to model: the answer to "what does this model read?" and "what breaks if this table changes?"

Where the week goes

• •Upstream changes break features without breaking pipelines. A renamed column, a unit change, a new category value or a null-filled default all keep the job green and the feature wrong.
• •Training-serving skew hides in plain sight. The offline feature and the online feature drift apart one fix at a time.
• •Training sets are hard to reproduce. Late-arriving corrections, changed label definitions and unversioned joins mean the dataset behind last quarter's model cannot be rebuilt as it was.
• •Drift alerts have no cause attached. The alert says a distribution moved. It does not say which source moved it.
• •Leakage is found late. A feature computed with data newer than the label looks like a great model until it meets production.
• •Contracts are agreements, not enforcement. The document exists; nothing stops a source team shipping a change that violates it.
• •Model reviews need evidence. Risk, audit and product review ask where the data came from, and the answer is assembled by hand.

Add it up and upkeep is most of the team's week. Here is how a typical week splits today, and once agents own the back office:

Time spent on new data products moves from 18% of the week to 48%. Most of the rest becomes review: people steer and approve agent work instead of doing it by hand.

Then and now

These are the impact metrics we move for AI and ML platform data teams:

• •Degraded model traced to its cause: from about two days to 30 min
• •Source schema change assessed against every feature: from about a day to 15 min
• •Point-in-time training set rebuilt: from about a day to 2 hr

Across the routine work in this domain:

The biggest change is assessing a source schema change: from about a day to 15 min. Work that waited in queues finishes in minutes or hours, because agents detect, fix and verify in one pass.

Use cases, in depth

An upstream rename that would have starved a feature

Today. A payments team renames a column and leaves the old one null. The pipeline succeeds, the null is filled with a default, and a discount feature collapses toward zero. The churn model keeps scoring. Nobody sees it until retention numbers move two weeks later, and the cause takes two days to find.

With Data Workers. The Schema Evolution agent flags the rename the day it ships. The Data Context & Catalog agent lists the feature views, training sets and models that read the column. The MLOps & Models agent compares feature statistics against the last release, and the Schema Evolution agent proposes a mapping and a bounded backfill for one approval.

What changes. The model never scores on the gap, and the source team gets a proposed contract line so the next change is visible before it ships.

A point-in-time-correct training set, rebuilt on demand

Today. The label definition changes, or a late correction lands on three months of events. Rebuilding the training set means re-deriving every feature as of each label date, by hand, while checking nothing newer than the label leaks in. Two engineers, most of a week, and no guarantee the result matches last time.

With Data Workers. The Data Context & Catalog agent traces every source behind each feature and label. The MLOps & Models agent rebuilds the features as of each label date. The Quality Monitoring agent adds the leakage and null-rate tests your team defines, and the MLOps & Models agent scores the candidate on the rebuilt set against the model in production.

What changes. A versioned dataset with lineage and a before-and-after score, ready for the model owner to review. Feast documents the concept of point-in-time joins; this is the work around it, done for every dataset.

Drift that comes with a cause attached

Today. The drift monitor fires on a ranking model. Is it real change in user behaviour, a broken upstream feed, or a pipeline that started dropping late events? The team opens five dashboards and a lineage diagram that was last updated a quarter ago.

With Data Workers. The MLOps & Models agent runs detect_model_drift and hands the finding to the Incident Debugging agent, which calls get_root_cause across the lineage to the source that moved. If a feed changed, the fix goes through the schema and quality path. If the world changed, retraining is proposed with the evidence.

What changes. Retraining happens when the data says the world moved, not when a pipeline broke, and every drift alert arrives with the likely cause and the models it touches. For the failure patterns in this space, see data quality for ML.

For teams serving LLM and agent workloads

The same platform increasingly feeds retrieval indexes, evaluation sets and agent tools. Those are data products too: they have sources that change, freshness that matters and lineage that a review will ask for. The swarm treats a retrieval corpus the way it treats a feature view, with an owner, a source map and checks. Teams that curate pretraining and fine-tuning corpora have a dedicated playbook in GenAI training data.

Measure the product, not just the pipeline

ML platform leaders are measured on how many models reach production and stay healthy there, not on pipeline counts. The Autopilot dashboard shows how much of the upkeep behind each model runs on its own and at what token cost, so the team can show leadership where its time now goes: new features, faster experiments and models that stay accurate longer.

Agents working together: A silent change, caught before the model feels it

Every ML platform team knows the failure that looks like nothing. A source team ships a routine change, every job turns green, and a model gets a little worse each day until a business metric tells you something is wrong. This is the same Thursday afternoon run by Data Workers.

The setup

A churn model and a pricing model read features built in dbt on Snowflake, orchestrated by Airflow and served from a feature store. A payments team owns one of the source tables. Autonomy in Autopilot:

• •Feature pipeline fixes and documentation: L2 today; a candidate for L3 on evidence.
• •Schema mapping that changes feature values: L2, held for the ML platform lead.
• •Model promotion and retraining: L0 to L2, always with a named approver.

14:02, the change ships

The payments team merges a release that renames the discount amount column to a minor-units version and leaves the old column null. No job fails.

14:04, detect and size the change

Schema Evolution agent detects the rename and the type change from the source. It flags that the old column will now return nulls.

Data Context & Catalog agent runs a blast-radius analysis from the column: three feature views, two training sets and four models read it, and two of the models are serving live traffic.

14:10, check the features

MLOps & Models agent compares feature statistics for the affected views against the last release. The 30-day average discount is already heading toward zero in the staging recompute. Quality Monitoring agent runs the null-rate and range checks the team defined and fails them in staging, before production features are overwritten.

14:15, propose the fix

Schema Evolution agent drafts the mapping from the new column back to the meaning the features expect, converting units, with a bounded backfill for the six hours already computed. It validates compatibility with the registered feature definitions and attaches a rollback point.

14:25, prove it

MLOps & Models agent scores the live candidate and the production model on the holdout set using the patched features. The difference sits inside the tolerance the model owner set.

14:40, one decision

Because the mapping changes feature values for live models, Autonomous Data-Conductor sends one decision to the ML platform lead: the diff, the affected models, the holdout comparison and the backfill window.

14:48, approved

The lead approves the mapping and the backfill. Conductor applies them, re-runs the checks on production features and publishes. The payments team receives a proposed data contract line: both columns present for a deprecation window.

What stays behind

• •Corrected features with tests that fail loudly on a null-filled default next time.
• •A receipt for every change: who proposed it, who approved it and how to roll it back.
• •Updated Spellbook pages for the three feature views, with owner, lineage and consumers.
• •A reusable routine. The next source change of this kind runs the same way, and the team can move the mapping step to L3 once the receipts support it.

Minute by minute

This is an illustrative example. The features are built in dbt on Snowflake under Airflow; two models score live traffic. Mappings that change feature values are at L2.

The agents on the job

Data Context Wizard holds one governed definition of each feature, label and metric, so the notebook, the training job and the serving path all mean the same thing by "30-day average discount". Autonomous Data-Conductor verifies the numbers that matter before a model retrains or a feature is published. Spellbook Data Catalog makes every feature, dataset and model input discoverable in plain English, with owner and lineage attached. See the MLOps & Models agent for how it guards the lifecycle.

What goes on autopilot

Once a domain has earned it, this work runs on its own, and each item removes a problem your team deals with today:

• •Upstream schema changes are mapped into feature pipelines the day they ship, so a Friday rename never becomes a Monday regression.
• •Feature freshness, null-rate and skew checks run every morning against the last release, so a feature that quietly went stale is caught before a model scores on it.
• •Feature, dataset and model-input documentation is generated and kept current, so a model review does not start with a week of archaeology.
• •Drift alerts arrive with a likely cause and the models they touch, so the first hour of an incident is not spent finding the source.

Where your people stay in charge

Agents never get blanket access. Each domain has its own autonomy level, set by you, and it only rises when the receipts show the agents are reliable. Everything starts at L1, observing only; reversible upkeep such as reruns, staging fixes, documentation and cleanup is usually first to run on its own.

• •Model promotion to production always goes to a named approver: it changes what customers experience, so the model owner approves with the evaluation, the data versions and the rollback point.
• •Retraining triggered by drift is proposed with the evidence of why the world moved, and the owner decides.
• •Access to labels and sensitive features stays with the data owner.
• •Label definitions and evaluation criteria stay entirely with people: they define what "good" means for the model.

Once a deployment has matured, here is how that ladder plays out across task classes:

Catalog and docs upkeep closes autonomously 88% of the time. For model promotion, autonomous closure is 0% by design: every promotion is proposed with the evidence and approved by a named person.

What happens when an agent is wrong

• •At L2 propose, nothing changes until a named person approves. The proposal shows the full diff, the affected models and how to roll it back. Unanswered requests expire and escalate.
• •L3 and L4 are where a domain can go next, on its own review record: acting on its own, with a rollback point on every change. Only the domain's owner raises its level, and if people start reverting the agents' changes, the domain pauses itself.
• •No agent approves or promotes its own work. Autonomy only grows on evidence, domain by domain, and repeated reverts pause the domain.
• •One org-wide stop halts all autonomous dispatch, and every action lands in a tamper-evident audit trail you can export with generate_audit_report.

How it fits your stack

Data Workers connects to what you already run: Snowflake, Databricks or BigQuery; dbt; Airflow, Dagster or Kubeflow; Feast, Tecton or a platform-native feature store; MLflow or a managed model registry; Kafka; your drift and serving monitors. Each agent is an MCP server, so the same agents work inside Claude Code, Cursor or Codex and in Slack or Teams through Spellbook. Nothing migrates.

• •Keep your feature store and registry. They define and serve features and record model versions. Data Workers keeps the tables behind them right. See how that works alongside Feast, MLflow and Amazon SageMaker.
• •Runs in your infrastructure with your credentials and your model key. The hosted Autonomous Data-Conductor sees workflow metadata only, never rows.
• •Uses the access you already govern. Agents act under scoped service identities and your existing roles; every grant goes through provision_access and is logged.
• •Proposes changes as diffs in your dbt project, orchestrator and warehouse, with blast radius, tests and a rollback point, and applies them only at the autonomy level you set for that domain.
• •Open-source core. The Apache-2.0 community edition ships 11 agents and 160+ MCP tools, including the ML tools above, so your engineers can read exactly what the agents call.

Governance and regulation

Models that score credit, insurance, hiring or health outcomes draw questions about their data. The EU AI Act sets data governance expectations for the training, validation and testing data of high-risk AI systems in Article 10 of Regulation (EU) 2024/1689, and the NIST AI Risk Management Framework asks organizations to map and document the data behind a system. Data Workers keeps lineage from source to feature to model current, scans training data for personal information with scan_pii, holds access to labels for the owner, and records every change in a tamper-evident audit trail. For the data-pipeline side of that work, see EU AI Act compliance for data pipelines.

Data Workers doesn't make you compliant; your controls, people and assessors do. It supplies controls and evidence wherever agents touch data. Not legal advice.

The alternatives AI and ML platform data teams weigh

Build it yourselves. A strong platform team can wire a coding agent to MCP servers for the warehouse, dbt and the registry. The work is everything around it: routing approvals to the right owner, blast radius across features and models, rollback, receipts, and current context about every other system. See why every MCP server for MLOps is read-only.

Feature stores, registries and MLOps platforms. They define, serve and record. They do not own the quality of the source tables beneath them or tell you which upstream change moved a model. Data Workers takes their signals, diagnoses, proposes the fix, verifies it and leaves the receipt.

Observability, contract and quality point tools. They alert on a failed check or guard the producer's pull request. Data Workers fixes what gets through, across systems. Contracts are most useful when something enforces them; see data contracts and AI agent enforcement.

In this space specifically. Model monitoring tools tell you a distribution moved; Data Workers tells you which source moved it and keeps the pipeline under the model correct.

The case for your CTO

What this replaces. Time lost to silently broken features, training sets that cannot be reproduced and drift alerts without a cause, plus the manual lineage work before every model review.

What your team gets back. An ML platform team that ships models, evaluation and new capabilities instead of finding out which table a drift alert came from.

Integration effort. Connecting is configuration, not a migration. The agents attach over MCP to the warehouse, orchestrator, dbt project, feature store and registry you already run, under the roles you already govern, inside your own infrastructure. Your engineers can read every tool the agents call in the open-source core.

Risk and control. Autonomy is set per domain. Everything starts at L1, observing only; reversible work moves up only when the receipts justify it; model promotion, retraining and access to sensitive features stay with a named approver; and one org-wide stop halts all autonomous work.

Why now. Every model, retrieval index and agent your company ships raises the cost of a wrong upstream value, and the fleet is growing faster than the team that guards its data.

The first win. Schema-change protection for the features behind one production model, at L2. The pilot below lays out the scope, what you see each week and the cost.

The climb: from assisted to autonomous to agentic

Our thesis is that the data platform climbs the way analytics did, in stages, and that the hard part of data work was never knowing what to do but doing it. For AI and ML platform data teams, the climb looks like this:

• •Assisted. Agents work alongside your team. They read the whole stack, show which features and models each upstream change touches and propose every fix with a receipt; model promotion and retraining always go to a named approver.
• •Autonomous, under your governance. Routine work runs on its own: upstream schema changes are mapped into feature pipelines the day they ship, freshness and skew are checked every morning, and documentation stays current. Your team governs the agents instead of doing the upkeep.
• •The agentic enterprise. An ML platform team that ships models and evaluation instead of chasing broken features. People set direction, and the platform carries it out.

You choose the altitude, every step is visible and reversible, and the pilot below is the first rung.

The pilot: how AI and ML platform data teams start

The pilot is how your team proves the value on its own data before committing.

• •What we connect. Your warehouse, dbt project, orchestrator, feature store and model registry, over MCP, inside your infrastructure. Nothing migrates.
• •Scope. Schema-change protection for the features behind one production model, at L2.
• •First week. Agents run at L1 across the scoped domain. They map every source, feature and lineage path to the model, and show the upstream changes they would have caught and the fixes they would have made, with receipts, so your team sees the value before anything changes.
• •By week two. Upstream schema changes are mapped into the feature pipelines and daily freshness and skew checks start running on reversible work. Feature value changes, model promotion and retraining stay at L2 with a named approver.
• •At day 30. We review the impact metrics against your baseline:

• •Degraded model traced to its cause: from about two days to 30 min
• •Source schema change assessed against every feature: from about a day to 15 min
• •Point-in-time training set rebuilt: from about a day to 2 hr
• •the share of catalog and docs upkeep closed autonomously, targeting 88% as the deployment matures

• •What you decide. Whether to expand to the next model or feature domain. The pilot fee is credited in full against the first year.
• •Cost. Start with a pilot ($7,500 one-time, credited in full against the first year). Scale is from $1,000 a month and Enterprise from $3,000 a month, billed annually, with unlimited seats, no usage meter and no markup on model spend. See pricing.

The pilot is the start of a longer curve:

By month 12, more than 42% of agent actions run fully autonomously and manual work falls below 10%, while model promotion and retraining stay with a named approver.

FAQ

Does it replace our feature store? No. Keep Feast, Tecton or your platform-native store for definitions and serving. Data Workers keeps the tables behind it right and tells you what an upstream change will touch.

Will agents promote or retrain models? Promotion is L2: proposed with the evaluation and the rollback point, and approved by the model owner. Retraining on drift is proposed with the evidence.

Can it catch feature leakage? It adds and runs the leakage and null-rate tests your team defines when it rebuilds a training set, and fails the build when they do not hold.

Can it ground our LLM and retrieval workloads? Yes. Retrieval corpora and evaluation sets are treated as governed data products with owners, source maps and checks.

Where do the agents run? In your infrastructure, with your credentials and model key. The hosted Conductor sees workflow metadata only.

Do we have to migrate anything? No. Your warehouse, dbt project, orchestrator, feature store, registry and monitoring stay where they are.

Sources

• •Data Workers pricing: https://dataworkers.io/pricing/
• •Feast, point-in-time joins: https://docs.feast.dev/getting-started/concepts/point-in-time-joins
• •Regulation (EU) 2024/1689 (the EU AI Act), Article 10 on data and data governance: https://eur-lex.europa.eu/eli/reg/2024/1689/oj
• •NIST, AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework