Best practices for agentic ML platform data: a field guide for ML platform teams

Ask an ML platform lead what the last bad week looked like and the answer is rarely a modelling problem. A source team renamed a column. The pipeline stayed green. A feature drifted to zero, the model kept scoring, and somebody noticed when a business metric moved two weeks later. The platform team owns the data under every production model, yet it does not own the systems that data comes from.

The mandate is to ship more models, on more workloads, with evidence that each one is built on data you can reproduce and explain. The trap is that, without automation, the team spends its week finding which table a drift alert came from and never builds what the scientists are waiting for.

Our view, as people who have run data platforms: the work that keeps features correct is repeatable, and agents should own it. This guide is for the AI and ML data teams that want to get there, and it is the practitioner's companion to the ML platform playbook.

The old world

It is 9:15 on a Monday. A payments team shipped a release on Thursday that renamed a discount column and left the old one null. Nothing failed, so nothing paged. The 30-day average discount feature sagged through the weekend and the churn model scored on the gap. A scientist notices a retention dip and asks whether the model got worse. The platform engineer opens five dashboards and a lineage diagram last edited a quarter ago. It takes two days to find the column.

Meanwhile, a risk reviewer wants to know what data went into last quarter's pricing model. Rebuilding the training set means re-deriving each feature as of each label date, by hand. Two engineers, most of a week, and no guarantee it matches the original.

The new world

Same Thursday with Data Workers. The Schema Evolution agent flags the rename within minutes. The Data Context & Catalog agent lists the three feature views, two training sets and four models that read the column. The MLOps & Models agent sees the discount feature collapsing in staging, and the Quality Monitoring agent fails the null-rate test before production features are overwritten. A mapping and a bounded backfill are proposed, and Autonomous Data-Conductor sends one decision to the ML platform lead, who approves it. The payments team gets a proposed contract line for next time.

Here is that Thursday as a chain of hand-offs between agents:

This report covers the seven practices that make it routine. At a glance:

The practices, one by one

1. Give every feature an owner, a source and an expected range

A feature nobody owns is a feature nobody notices going wrong. An owner, a named source and an expected range turn "the number looks odd" into a check that can fail. A feature store serves features but does not make sure each one carries that context.

Why it's harder than it looks: the definitions were written by whoever shipped the first model, and "30-day average discount" lives in three slightly different SQL files. Documentation drives go stale in a month.

1. 1.The Data Context & Catalog agent inventories feature views and calls generate_documentation for each, with source, owner and consumers.
2. 2.It calls mark_authoritative and define_business_rule for the definitions the team confirms, so notebook, training job and serving path share one meaning.
3. 3.The Quality Monitoring agent calls set_sla for freshness and create_quality_tests_for_pipeline for range and null-rate checks.
4. 4.The ML platform lead approves any change to a definition that live models read.

2. Map upstream schema changes into features the day they ship

Most feature failures begin outside the ML platform. A renamed column, a unit change or a null-filled default keeps the job green and the feature wrong. This is where we would start; see schema changes that never break downstream.

Why it's harder than it looks: the change comes from a team that has never heard of your feature. A contract document exists, but nothing stops a violating change from merging.

1. 1.The Schema Evolution agent calls detect_schema_change and assess_impact on the source and lists the readers.
2. 2.The Data Context & Catalog agent calls blast_radius_analysis: which feature views, training sets and models read the column, and which serve live traffic.
3. 3.The Schema Evolution agent calls check_compatibility and generate_migration for the mapping and a bounded backfill, with a rollback point.
4. 4.The MLOps & Models agent calls get_feature_stats on the affected views and compares them with the last release, in staging.
5. 5.A mapping that changes feature values for a live model goes to the ML platform lead as one decision with the full diff. The source team receives a proposed contract line.

Across the routine work these first practices cover, here is the difference in elapsed time:

The biggest change is assessing a source schema change: from about a day to 15 min. Work that waited in queues finishes in minutes or hours, because agents detect, fix and verify in one pass.

3. Check freshness, null rates and skew every morning

A feature can be stale, half-null or different between training and serving without any job failing. Data quality for ML lists the failure patterns, and almost all are quiet.

Why it's harder than it looks: thresholds hand-written for hundreds of feature views rot, so people mute the alerts, and nobody owns the offline-versus-online comparison.

1. 1.A daily routine starts the Quality Monitoring agent, which calls run_quality_check on every feature view and get_anomalies for deviations from each view's normal behaviour.
2. 2.The Data Context & Catalog agent calls check_freshness on the sources behind the views.
3. 3.The MLOps & Models agent calls get_feature_stats against the last release, so a shift shows up as a difference.
4. 4.A late feature is rerun on reversible work. A feature whose values changed is held for its owner, with the evidence.

4. Rebuild training sets as of each label date, and test for leakage

A result from March has to be rerun in October, and a feature computed with data newer than the label looks like a great model until it meets production. Feast documents point-in-time joins; the work is doing it, with tests, for every dataset.

Why it's harder than it looks: late-arriving corrections, a changed label definition and unversioned joins all alter history. The rebuild is a week of careful work that gets skipped, so the dataset behind the production model quietly stops being reproducible.

1. 1.The Data Context & Catalog agent calls trace_cross_platform_lineage to find every source behind each feature and label.
2. 2.The MLOps & Models agent calls create_feature_pipeline to rebuild each feature as of each label date and version the result.
3. 3.The Quality Monitoring agent calls create_quality_tests_for_pipeline for the leakage and null-rate tests the team defines.
4. 4.The MLOps & Models agent calls evaluate_model to score the candidate on the rebuilt set against the production model.
5. 5.The model owner reviews the dataset, its lineage and the before-and-after score. Label definitions stay with people.

5. Trace every drift alert to a cause before anyone retrains

A drift alert says a distribution moved, not whether the world changed or a feed broke, and the two need different responses. Retraining on a broken feed bakes the bug into the next model.

Why it's harder than it looks: the cause sits two or three hops upstream, across tools with separate lineage. The first hour of every incident goes on finding the source.

1. 1.The MLOps & Models agent calls detect_model_drift and hands the finding to the Incident Debugging agent.
2. 2.The Incident Debugging agent calls diagnose_incident and get_root_cause across the lineage, back to the change that moved the feature.
3. 3.If a feed changed, the fix runs through practice 2. If the world changed, the agent assembles the evidence for retraining.
4. 4.The model owner decides whether to retrain, with the evidence in front of them.

6. Put promotion, retraining and label access behind a named approver

Promoting a model changes what customers experience, and labels and sensitive features can carry personal or licensed content. Trust depends on a person owning those decisions.

Why it's harder than it looks: the choice is usually binary. Either every change waits for a review nobody has time for, or changes go straight to production and get rolled back after the complaints.

1. 1.The MLOps & Models agent assembles the promotion package with evaluate_model and get_model_versions: the data, code and feature versions behind the candidate.
2. 2.The Data Access & Governance agent calls check_policy, scan_pii and provision_access for label and feature requests, and logs every grant.
3. 3.Autonomous Data-Conductor routes one decision to the right owner, with the diff, the evaluation and the rollback point, and escalates what goes unanswered.
4. 4.The model owner approves promotion and retraining. The data owner approves access to labels and sensitive features. No agent approves its own work.

The balance, autonomy where the work is reversible and a named approver where it matters, looks like this by task class:

Catalog and docs upkeep closes autonomously 88% of the time. For model promotion, autonomous closure is 0% by design: every promotion is proposed with the evidence and approved by a named person.

7. Keep lineage from source to feature to model ready for the review

Risk, audit and product reviews all ask where a model's data came from. The EU AI Act sets data governance expectations for training, validation and testing data in Article 10 of Regulation (EU) 2024/1689, and the NIST AI Risk Management Framework asks organizations to map and document the data behind a system. Lineage is the shared answer; see end-to-end lineage.

Why it's harder than it looks: lineage drawn once goes stale within a release, and the warehouse, dbt, the feature store and the registry each hold part of the picture.

1. 1.The Data Context & Catalog agent calls get_lineage and trace_cross_platform_lineage to keep source, feature and model joined as pipelines change.
2. 2.The MLOps & Models agent calls register_model, so every registered version carries its data and feature versions.
3. 3.The Data Access & Governance agent calls generate_audit_report for the evidence pack, and scan_pii on training data.
4. 4.The model owner reviews the pack before it goes out. Data Workers supplies evidence; your controls and assessors decide compliance.

Designing the autonomy ladder

Autonomy is set per task class, not per tool, and only a domain's owner can raise it. Everything starts at L1, observing only. L0 to L2 run today: watch, suggest, or act after approval. L3 and L4 are where a domain can go on evidence from its own receipts, never by default. This is how we would set each class:

The operating model: who owns what

Agents change who does the work, not who is accountable for it:

The swarm behind the practices

No single agent delivers these practices. Each one is a hand-off between specialists, coordinated under the autonomy you set per domain.

Data Context Wizard holds one governed definition of each feature, label and metric, so the notebook, the training job and the serving path all mean the same thing. Autonomous Data-Conductor verifies the numbers that matter before a model retrains or a feature is published. Spellbook Data Catalog makes every feature, dataset and model input discoverable in plain English, with owner and lineage attached. For the lifecycle agent, see inside the MLOps & Models agent.

What disappears

The clearest sign the practices are working is what stops happening:

• •The Monday regression after a source team's Thursday release.
• •Two days spent finding which table a drift alert came from.
• •Training sets that cannot be rebuilt as they were.
• •Leakage discovered after the model meets production.
• •A week of archaeology before every model review.
• •Retraining on a broken feed.

The back office keeps running. Changes are caught where they enter, not where a model reads them. New features get tests and documentation without anyone writing them. Fixes are rerun, verified and explained before the scientist opens a dashboard, and every action leaves a signed receipt, so the team can see what happened and roll it back.

An ML platform lead's scorecard

These are targets to set against your own baseline, not results:

• •Time from source schema change to assessed impact: target 15 min.
• •Time from degraded model to its cause: target 30 min.
• •Time to rebuild a point-in-time training set: target 2 hr.
• •Share of feature views with an owner, source, range and freshness SLA: target all production features.
• •Share of production models whose full lineage can be produced on request: target all.
• •Share of upkeep closed autonomously, and its token cost: target 88% for catalog and docs upkeep as the deployment matures.

Where this goes next

The same practices apply to the retrieval indexes, evaluation sets and agent tools your platform increasingly feeds: they have sources that change and lineage a review will ask for. Teams that curate pretraining and fine-tuning corpora have a dedicated field guide in GenAI training data.

The climb: from assisted to autonomous to agentic

Our thesis is that the data platform climbs the way analytics did, in stages, and that the hard part of data work was never knowing what to do but doing it. For ML platform data teams, the climb looks like this:

• •Assisted. Agents work alongside your team. They read the whole stack, show which features and models each upstream change touches and propose every fix with a receipt; promotion and retraining always go to a named approver.
• •Autonomous, under your governance. Routine work runs on its own: schema changes are mapped into feature pipelines the day they ship, freshness and skew are checked every morning, and documentation stays current. Your team governs the agents instead of doing the upkeep.
• •The agentic enterprise. An ML platform team that ships models and evaluation instead of chasing broken features. People set direction, and the platform carries it out.

You choose the altitude, every step is visible and reversible, and the pilot below is the first rung.

The pilot: how ML platform data teams start

The pilot is how your team proves the value on its own data before committing.

• •What we connect. Your warehouse, dbt project, orchestrator, feature store and model registry, over MCP, inside your infrastructure. Nothing migrates; teams on Feast keep it.
• •Scope. Schema-change protection for the features behind one production model, at L2.
• •First week. Agents run at L1 across the scoped domain. They map every source, feature and lineage path to the model, and show the upstream changes they would have caught and the fixes they would have made, with receipts, so your team sees the value before anything changes.
• •By week two. Upstream schema changes are mapped into the feature pipelines and daily freshness and skew checks start running on reversible work. Feature value changes, promotion and retraining stay at L2 with a named approver.
• •At day 30. We review the impact metrics against your baseline:

• •Degraded model traced to its cause: from about two days to 30 min
• •Source schema change assessed against every feature: from about a day to 15 min
• •Point-in-time training set rebuilt: from about a day to 2 hr
• •the share of catalog and docs upkeep closed autonomously, targeting 88% as the deployment matures

• •What you decide. Whether to expand to the next model or feature domain. The pilot fee is credited in full against the first year.
• •Cost. Start with a pilot ($7,500 one-time, credited in full against the first year). Scale is from $1,000 a month and Enterprise from $3,000 a month, billed annually, with unlimited seats, no usage meter and no markup on model spend. See pricing.
• •The guarantee. In a 90-day pilot, ten engineering hours per data engineer per week are recovered against a baseline both sides sign on day one, or the $7,500 fee is refunded.

The pilot is the start of a longer curve:

By month 12, more than 42% of agent actions run fully autonomously and manual work falls below 10%, while promotion and retraining stay with a named approver.