Best practices for agentic GenAI training data: a field guide for data curation and eval teams
Model quality is now a data engineering problem. This report covers how training data teams keep data traceable, clean and affordable with agents.
Every company shipping generative models now has a training data team, whether at a frontier lab, a chip and platform company, or a creative software company training its own models. Their outputs, curated corpora, synthetic data, human feedback and eval sets, decide model quality.
The work is data engineering at enormous scale with unusual constraints: rights must be tracked to the asset, filters versioned per model, eval sets kept clean, and storage kept from exploding as every experiment copies the corpus.
Agents let training data teams scale the corpus while keeping every asset traceable from license to model.
The old world
A rights holder narrows its license to exclude one product line. The team spends three weeks finding which assets, datasets and mixes are affected and which model versions trained on them. Meanwhile an eval set leaked into a training mix last month, and the benchmark gains on the latest model are partly fake.
The new world
With Data Workers, Access & Governance found every affected dataset and mix in minutes, Context Wizard listed the model versions that trained on them, Pipeline Building built clean replacements and Conductor held the swap for the training data lead. A contamination check runs before every training run, so eval sets stay clean.
Here is that new world as a chain of hand-offs between agents:
The full playbook walks through this scenario and three more use cases in depth.
The rest of this report walks through the 6 practices that make that morning possible. At a glance:
The practices, one by one
1. Track rights to the asset
Every asset needs a source, a license and a usage scope.
The old world. Spreadsheets.
The new world. Access & Governance enforces license scope on datasets and training jobs.
In practice. A license change is enforced across every mix in hours.
What to measure. Assets with rights on record.
2. Version every filter
You should know which filter version every model saw.
The old world. Reconstructed from notebooks.
The new world. Pipeline Building versions filters and Context Wizard links them to model versions.
In practice. A safety filter change is traceable to every downstream run.
What to measure. Runs with full data lineage.
Practices like these compound. Across the routine work they cover, here is the difference in elapsed time:
The biggest change is License-change impact found: from about 2 weeks to minutes. Work that waited in queues finishes in minutes or hours, because agents detect, fix and verify in one pass.
3. Check eval contamination before every run
Leaked test data makes scores lie.
The old world. Found after scores look too good.
The new world. Conductor runs a contamination check before each run and blocks overlap.
In practice. A leaked eval subset is blocked before training.
What to measure. Runs with contamination checked.
4. Classify sensitive content on arrival
PII and unsafe content should never reach a mix.
The old world. Filtered late.
The new world. Data Security classifies content as it lands.
In practice. PII in a new dataset is flagged before it is used.
What to measure. Sensitive content reaching mixes.
5. Measure label quality continuously
Human-feedback data degrades quietly.
The old world. Spot checks.
The new world. Quality Monitoring tracks agreement and labeler quality by task.
In practice. A drop in expert agreement is caught on the first batch.
What to measure. Batches gated on quality.
That balance, autonomy where work is reversible and a named approver where it matters, shows up clearly by task class:
Storage cleanup closes autonomously 88% of the time. For Released mixes and eval sets, autonomous closure is 0% by design: every change is proposed with a full diff and approved by a named person.
6. Stop copying the corpus
Duplicated data is the biggest cost driver.
The old world. One copy per experiment.
The new world. Cost Savings & Data Cleanup retires stale copies and tiers storage.
In practice. Corpus storage halves while experiments grow.
What to measure. Storage per unique token.
Designing the autonomy ladder
Autonomy is set per task class, not per tool. This is where we would set each class for a mature deployment, and why:
| Task class | Closed autonomously | Escalated | Handled by a person | Where we set it | Why |
|---|---|---|---|---|---|
| Storage cleanup | 88% | 8% | 4% | L4 once the receipts are clean | The work is reversible and repeatable |
| Contamination checks | 84% | 12% | 4% | L4 once the receipts are clean | The work is reversible and repeatable |
| Sensitive-content classification | 70% | 22% | 8% | L3: acts and keeps a rollback point | Most cases are routine; the rest escalate with evidence |
| Dedup and filter reruns | 72% | 20% | 8% | L3: acts and keeps a rollback point | Most cases are routine; the rest escalate with evidence |
| License-scope enforcement | 62% | 30% | 8% | L3: acts and keeps a rollback point | Most cases are routine; the rest escalate with evidence |
| Released mixes and eval sets | 0% | 88% | 12% | L2: always proposed to a named approver | The number carries accountability outside the data team |
The operating model: who owns what
Agents change who does the work, not who is accountable for it:
| Role | Owns | What the agents take off their plate |
|---|---|---|
| Principal PM for GenAI Training Data | Standards, autonomy levels per domain and which domain goes next | Assembling status, chasing owners, keeping the catalog current |
| Domain owners and approvers | Changes to released training mixes and eval sets | Preparing the diff, blast radius and rollback plan for every proposal |
| Data engineers | Reviewing agent proposals and building new data products | Every training run is checked for eval contamination before it starts; license scope is enforced on every dataset and job |
The swarm behind the practices
No single agent delivers these practices. Each one is a hand-off between specialists, coordinated under the autonomy you set per domain.
| Agent | What it does for training data teams |
|---|---|
| Data Ingestion and Connectors | Land licensed, owned, synthetic and labeled data from every source and profile it |
| Data Access & Governance | Enforces license and usage scope on every dataset and training job |
| Data Security | Classifies PII and sensitive content before it enters any training mix |
| Pipeline Building | Builds versioned filter, dedup and mixing pipelines with tests |
| Quality Monitoring | Tracks filter pass rates, label agreement, duplicate rates and eval contamination |
| Cost Savings & Data Cleanup | Retires stale corpus copies and right-sizes storage tiers |
Context Wizard links every dataset version to its sources, licenses, filters and the models that trained on it. Conductor verifies that eval sets stay clean before every training run. Spellbook gives researchers a searchable, governed catalog of datasets with rights and lineage on every page.
What disappears
The clearest sign the practices are working is what stops happening:
- •Weeks spent tracing a license change.
- •Leaked eval sets inflating scores.
- •Untraceable filter versions.
- •PII discovered after training.
- •Silent label-quality decline.
- •A full corpus copy per experiment.
The magic is not any single agent. It is that the back office simply keeps running. Bad data is caught where it enters, not where it is read. Every new table gets quality rules and documentation without anyone writing them. Access follows policy instead of a queue. Fixes are rerun, verified and explained before anyone opens a dashboard. Cost is cleaned up every week instead of every budget cycle. And every one of those actions leaves a signed receipt, so the team can see exactly what happened and roll anything back. For the people on the team, the experience is quieter mornings, fewer tickets and more time for the products the business asked for.
A product manager's scorecard
If you own this data as a product, these are the numbers to bring to every quarterly review:
- •Training runs with full lineage
- •Runs with contamination checks
- •Assets with rights on record
- •Time to trace a license change
- •Storage per unique token
- •Share of upkeep autonomous
The climb: from assisted to autonomous to agentic
Our thesis is that the data platform climbs the way analytics did, in stages, and that the hard part of data work was never knowing what to do but doing it. For GenAI training data teams, the climb looks like this:
- •Assisted. Agents work alongside your team. They read the whole stack, show what needs doing and propose every fix with a receipt; changes to released training mixes and eval sets always go to a named approver.
- •Autonomous, under your governance. Routine work runs on its own: every training run is checked for eval contamination before it starts, license scope is enforced on every dataset and job, and stale corpus copies are retired and storage tiered. Your team governs the agents instead of doing the upkeep.
- •The agentic enterprise. A training data team that can scale the corpus and still answer 'what did this model train on' in minutes. People set direction, and the platform carries it out.
You choose the altitude, every step is visible and reversible, and the pilot below is the first rung.
The pilot: how GenAI training data teams start
The pilot is how your team proves the value on its own data before committing.
- •What we connect. Object storage; Spark or Ray processing; labeling tools; training run trackers; eval stores; a catalog, over MCP, inside your infrastructure. Nothing migrates.
- •Scope. Contamination and license checks on one training mix.
- •First week. Agents run at L1 across the scoped domain. They map every pipeline, definition and lineage path and show the fixes they would have made, with receipts, so your team sees the value before anything changes.
- •By week two. Every training run is checked for eval contamination before it starts and license scope is enforced on every dataset and job start running on reversible work. Changes to released training mixes and eval sets stay at L2 with a named approver.
- •At day 30. We review the impact metrics against your baseline:
- •License-change impact found: from about 2 weeks to minutes
- •Finding the right dataset: from about 8 hours to 30 s
- •New dataset classified: from about a week to 15-20 min
- •the share of storage cleanup closed autonomously, targeting 88% as the deployment matures
- •What you decide. Whether to expand to the next domain. The pilot fee is credited in full against the first year.
- •Cost. Start with a pilot ($7,500 one-time, credited in full against the first year). Scale is from $1,000 a month and Enterprise from $3,000 a month, billed annually, with unlimited seats, no usage meter and no markup on model spend. See pricing.
The pilot is the start of a longer curve:
By month 12, more than 46% of agent actions run fully autonomously and manual work falls below 10%, while released training mixes stay with a named approver.
Read the full playbook: Training data, human data and eval teams