Best practices for agentic physical AI data: a field guide for robotics and perception data teams
Physical AI learns from fleet logs, teleoperation and labels. This report covers how robotics and perception data teams turn raw drives into training data faster with agents.
Robots and autonomous vehicles generate data faster than any team can curate it. Fleet logs, teleoperation sessions and vendor labels pile up in object storage, and the bottleneck is finding the right minutes, converting them and labeling them well.
Physical AI data teams sit between research, data engineering and labeling operations. Every new robot platform adds a format, every label spec change risks vendor drift, and coverage questions take days to answer.
Agents turn that pipeline into something continuous: logs indexed as they land, formats standardized automatically, label batches gated on quality, rare-scenario sets built on request.
The old world
Research needs rain and night-time left turns. An engineer spends a week searching drives. Meanwhile a labeling vendor missed a spec change, pedestrian labels at night degraded, and two batches already reached the training set.
The new world
With Data Workers, scenario attributes are indexed as logs land, so the rare-scenario set is built as a mission in a day. Quality Monitoring caught the label drop on the first batch, Incident Debugging tied it to the spec change and the batch was quarantined before it reached training.
Here is that new world as a chain of hand-offs between agents:
The full playbook walks through this scenario and three more use cases in depth.
The rest of this report walks through the 6 practices that make that morning possible. At a glance:
The practices, one by one
1. Index scenarios on ingestion
Coverage questions should take one query.
The old world. Days of searching logs.
The new world. Data Context & Catalog indexes scenario attributes as logs land.
In practice. How many night-time left turns in rain? Answered in seconds.
What to measure. Time to answer a coverage question.
2. Standardize formats across platforms
Every new robot means a new converter.
The old world. Hand-written converters.
The new world. Schema Evolution maps new sensor and label formats to the standard.
In practice. A new gripper platform produces usable data on day one.
What to measure. Days to onboard a platform.
Practices like these compound. Across the routine work they cover, here is the difference in elapsed time:
The biggest change is Bad label batch quarantined: from about 3 days to 8-15 min. Work that waited in queues finishes in minutes or hours, because agents detect, fix and verify in one pass.
3. Gate label batches on quality
Bad labels poison training sets.
The old world. Spot checks after the fact.
The new world. Quality Monitoring checks agreement per batch and quarantines failures.
In practice. A degraded batch never reaches training.
What to measure. Bad batches reaching training.
4. Version training sets with their sources
Every set needs logs, spec and quality report.
The old world. Inconsistent.
The new world. Autopilot enforces versioning as a standard.
In practice. Every set is reproducible.
What to measure. Sets fully versioned.
5. Build sets as missions
Rare-scenario sets are repeated work.
The old world. A week per set.
The new world. Autopilot builds scenario sets from the fleet archive.
In practice. A new set in a day.
What to measure. Days per scenario set.
That balance, autonomy where work is reversible and a named approver where it matters, shows up clearly by task class:
Storage tiering closes autonomously 88% of the time. For Released training sets, autonomous closure is 0% by design: every change is proposed with a full diff and approved by a named person.
6. Tier raw logs
Storage grows with every drive.
The old world. Kept hot.
The new world. Cost Savings & Data Cleanup tiers logs no training set uses.
In practice. Storage cost falls as the fleet grows.
What to measure. Storage cost per fleet hour.
Designing the autonomy ladder
Autonomy is set per task class, not per tool. This is where we would set each class for a mature deployment, and why:
| Task class | Closed autonomously | Escalated | Handled by a person | Where we set it | Why |
|---|---|---|---|---|---|
| Storage tiering | 88% | 8% | 4% | L4 once the receipts are clean | The work is reversible and repeatable |
| Ingestion verification | 82% | 12% | 6% | L4 once the receipts are clean | The work is reversible and repeatable |
| Label batch gating | 74% | 18% | 8% | L3: acts and keeps a rollback point | Most cases are routine; the rest escalate with evidence |
| Format conversions | 64% | 26% | 10% | L3: acts and keeps a rollback point | Most cases are routine; the rest escalate with evidence |
| Scenario indexing | 80% | 14% | 6% | L4 once the receipts are clean | The work is reversible and repeatable |
| Released training sets | 0% | 88% | 12% | L2: always proposed to a named approver | The number carries accountability outside the data team |
The operating model: who owns what
Agents change who does the work, not who is accountable for it:
| Role | Owns | What the agents take off their plate |
|---|---|---|
| Perception Data Product Leader | Standards, autonomy levels per domain and which domain goes next | Assembling status, chasing owners, keeping the catalog current |
| Domain owners and approvers | Changes to released training sets and label specs | Preparing the diff, blast radius and rollback plan for every proposal |
| Data engineers | Reviewing agent proposals and building new data products | Scenario attributes are indexed as logs land; every label batch is gated on agreement and quality |
The swarm behind the practices
No single agent delivers these practices. Each one is a hand-off between specialists, coordinated under the autonomy you set per domain.
| Agent | What it does for physical AI data teams |
|---|---|
| Data Ingestion and Connectors | Land fleet logs, teleoperation sessions and vendor labels and index them |
| Schema Evolution | Maps new sensor, robot and label formats into the standard |
| Pipeline Building | Builds curation, sampling and conversion pipelines with tests |
| Quality Monitoring | Label agreement, vendor quality, sensor gaps and calibration checks |
| Data Context & Catalog | Indexes scenarios and attributes so coverage questions take one query |
| Cost Savings & Data Cleanup | Tiers and retires raw logs that no training set uses |
Context Wizard makes coverage questions answerable in plain English with the source logs attached. Conductor keeps labeling and curation pipelines on schedule and verified. Spellbook gives research a governed catalog of scenarios, datasets and labels.
What disappears
The clearest sign the practices are working is what stops happening:
- •Week-long log searches.
- •Hand-written converters for every platform.
- •Bad label batches in training sets.
- •Unreproducible training sets.
- •Coverage guesswork.
- •Raw logs kept hot forever.
The magic is not any single agent. It is that the back office simply keeps running. Bad data is caught where it enters, not where it is read. Every new table gets quality rules and documentation without anyone writing them. Access follows policy instead of a queue. Fixes are rerun, verified and explained before anyone opens a dashboard. Cost is cleaned up every week instead of every budget cycle. And every one of those actions leaves a signed receipt, so the team can see exactly what happened and roll anything back. For the people on the team, the experience is quieter mornings, fewer tickets and more time for the products the business asked for.
A product manager's scorecard
If you own this data as a product, these are the numbers to bring to every quarterly review:
- •Time to answer a coverage question
- •Days to build a scenario set
- •Bad batches reaching training (target zero)
- •Sets fully versioned
- •Storage per fleet hour
- •Share of upkeep autonomous
The climb: from assisted to autonomous to agentic
Our thesis is that the data platform climbs the way analytics did, in stages, and that the hard part of data work was never knowing what to do but doing it. For physical AI, robotics and perception data teams, the climb looks like this:
- •Assisted. Agents work alongside your team. They read the whole stack, show what needs doing and propose every fix with a receipt; changes to released training sets and label specs always go to a named approver.
- •Autonomous, under your governance. Routine work runs on its own: scenario attributes are indexed as logs land, every label batch is gated on agreement and quality, and new sensor and robot formats are mapped to the standard. Your team governs the agents instead of doing the upkeep.
- •The agentic enterprise. Researchers who get the training sets they ask for in days. People set direction, and the platform carries it out.
You choose the altitude, every step is visible and reversible, and the pilot below is the first rung.
The pilot: how physical AI, robotics and perception data teams start
The pilot is how your team proves the value on its own data before committing.
- •What we connect. Fleet log storage; ROS or MCAP formats; labeling vendors; Spark or Ray; training pipelines, over MCP, inside your infrastructure. Nothing migrates.
- •Scope. Label gating on one vendor.
- •First week. Agents run at L1 across the scoped domain. They map every pipeline, definition and lineage path and show the fixes they would have made, with receipts, so your team sees the value before anything changes.
- •By week two. Scenario attributes are indexed as logs land and every label batch is gated on agreement and quality start running on reversible work. Changes to released training sets and label specs stay at L2 with a named approver.
- •At day 30. We review the impact metrics against your baseline:
- •Coverage question answered: from about 3 days to seconds
- •Bad label batch quarantined: from about 3 days to 8-15 min
- •Raw logs tiered: from about a month to hours
- •the share of storage tiering closed autonomously, targeting 88% as the deployment matures
- •What you decide. Whether to expand to the next domain. The pilot fee is credited in full against the first year.
- •Cost. Start with a pilot ($7,500 one-time, credited in full against the first year). Scale is from $1,000 a month and Enterprise from $3,000 a month, billed annually, with unlimited seats, no usage meter and no markup on model spend. See pricing.
The pilot is the start of a longer curve:
By month 12, more than 48% of agent actions run fully autonomously and manual work falls below 10%, while released training sets stay with a named approver.
Read the full playbook: Robotics, autonomy and perception data teams