How Large Companies Evaluate Data Agents in Production

Public benchmarks tell you how capable a model is on someone else's tasks. They cannot tell you whether an agent gets your revenue definition right, picks the right one of your four order tables, or respects who is allowed to see what. Every company that has put a data agent in front of its own employees has had to build that evaluation itself, and eight of them - OpenAI, DoorDash, Hex, Uber, Meta, Anthropic, Notion and Red Hat - have written up how.

Read side by side, the write-ups agree on more than they differ. This report draws out the five patterns they share: where the eval set comes from, how answers are graded, what gets measured, why most of the accuracy work turns out to be context work, and where a person stays in the loop. For the benchmark side of the picture, see our Data Agent Evals Report 2026.

• •The eval set is mined from production. Real questions, past decisions and query logs, not test cases written from scratch.
• •Graders check the path as well as the answer. Golden SQL and result data first, then rubrics for tool use, table choice and process.
• •Agreement with people is a metric. The strongest numbers in the set measure how often the agent matches a decision a human already made.
• •Accuracy is mostly a context problem. The biggest gain reported anywhere in the set came from better context, not a better model, and every team that keeps context fresh uses agents to do it.
• •The human is placed by consequence, not by volume. No team reviews every answer; most put a person at one specific point where a mistake is costly.

Where do production eval sets come from?

Almost none of these teams sat down and wrote an eval set. They harvested one from work their company was already doing.

DoorDash has the largest: more than 1,900 questions across 19 business domains for its Vera agent. A frontier model reads Slack threads and Jira tickets and drafts a full eval proposal for each candidate question; after automatic de-duplication, domain owners accept about 95% of them. Meta went further and used decisions rather than questions, replaying historical data-access requests against the outcome a person had chosen. Red Hat starts from historical SQL query logs, and Uber re-runs historical queries to see whether answers drift over time.

OpenAI is the closest to a hand-built set: curated question-and-answer pairs, each with a manually written "golden" SQL query. Even there, the questions come from how the company's 3,500+ internal users actually work with more than 600 petabytes of data in 70,000 datasets.

The one deliberate exception proves the point. Hex built Shoebox, a lab whose test bed is Shorelane Commerce, a synthetic business with six years of data and about 30,000 hand-crafted lines of data generators, dbt models, documentation and personas. It is synthetic so that Hex can control the ambiguity - five different columns could plausibly be called "revenue" - and frame every eval as a pairwise experiment between a candidate run and a baseline. A vendor shipping to many customers cannot mine one company's Slack, so it builds a company instead.

How do production teams grade a data agent?

The graders fall on a spectrum from fully deterministic to fully judged, and the teams with the most at stake sit nearest the deterministic end.

OpenAI compares both the generated SQL and the data it returns against the golden query and feeds those signals into its Evals grader; Uber's Finch is checked against golden queries the same way. DoorDash uses an LLM judge, but anchors it to specifics: each answer is scored out of 3.0 on tool usage, table selection and answer correctness. Hex scores runs against rubrics for named behaviours such as ToolEfficiency, SemanticLayerUsage and WorkspaceGuideAdherence.

Scoring behaviour, not just the final number, is what lets a team see that an agent got the right answer the wrong way - and a wrong way that happens to work today is tomorrow's incident. Two teams also put a checker inside the agent itself: Anthropic runs a sql-reviewer sub-agent on every query before the final answer, and Meta's analytics agent has a separate model check output against guardrail recipes such as "WAU should be < 8 billion" and "Always filter by is_test=false".

Evals also run continuously rather than at release. OpenAI runs its evals during development "like unit tests" and as canaries in production. DoorDash found that the tail, not the median, drives timeouts and partial results, and counts a timeout as a failure.

Which metrics do production teams actually use?

Across the eight, the metrics fall into four groups, and the strongest setups use more than one:

The third group is the least common and the most telling. Meta measured how often its access agent reached the same decision a person had, and reported the two error directions separately, because for an access agent they are not equal.

Why is context the main lever on accuracy?

The largest improvement reported anywhere in the set did not come from a model upgrade. Anthropic's unguided agent "didn't exceed 21%" on its internal analytics evals; with skills - a knowledge router to domain reference files and a runbook that "encodes the process a senior analyst would follow" - it is "consistently above 95% in aggregate". Anthropic names the three failure modes the work targets: concept-to-entity ambiguity, data staleness and retrieval failure.

OpenAI tells the same story from the other side: most of the engineering effort on its agent went into context, which it describes in six layers, from table usage and human annotations through institutional knowledge and memory to runtime queries against the warehouse. Meta's analytics agent builds a profile from the tables an analyst queried in the past 90 days, so you can "clone" a colleague by pointing it at their query history.

Once context drives accuracy, stale context becomes the main way accuracy decays, and the teams that address it use agents to maintain context, with people on the ambiguous cases:

• •Anthropic keeps skill files next to the reporting models, flags any reporting-model change that does not touch a skill file in code review, and runs a scheduled agent that scans stakeholder channels for correction language and proposes fixes.
• •Notion runs a review agent over conversations, Slack, Notion and GitHub for signs that context is out of date, and a triage agent that applies small, well-supported fixes and sends ambiguous ones to a person. Its Data Scout is used by about half of Notion's employees, and weekly requests to the data team fell by about 80% in two months.
• •Red Hat's agent drafts guidance from historical query patterns, and data product owners review and refine it. The eval question shifts from "is this answer right" to "is the context the agent runs on right".

How should human-in-the-loop work for a data agent?

None of these teams reviews every answer by hand, and none lets the agent run unsupervised where it matters. As the chart at the top shows, the human sits in one of four places:

• •Before a costly step. For DoorDash's hardest questions - causal analysis, experiment readouts, committed forecasts - Vera writes an explicit plan of timeframes, metrics, scope and constraints, and nothing executes until the user approves or adjusts it.
• •On the decision that cannot be undone. Meta's data-owner agent works "like a junior engineer who assists the team with security tasks", but the access decision stays with the owner.
• •On the context. Anthropic, Notion and Red Hat route corrections to the people who own the definitions, so a fix lands once and every later answer benefits.
• •In the permissions. OpenAI's agent is pass-through - "users can only query tables they already have permission to access" - and Notion has each user sign in to Snowflake with their own credentials.

Uber has described a "Request validation" button that routes a Finch answer to a subject-matter expert, but as upcoming rather than shipped. The common thread is that a person is placed where a mistake is expensive, not spread across every answer. That is the same principle we build around: an agent that can change your data should have to pass its evaluation and its approvals in the path the change travels, not in a review queue someone checks later.

The eight write-ups at a glance

How Data Workers builds this in

Each of these teams built its eval system around one internal agent. We build the Autonomous Agentic Data Platform, a swarm of 20 specialist agents, so the same practices have to hold for every agent and every customer stack. Here is how they show up in the product.

• •An eval set per agent, with a held-out slice. Each agent is scored against a versioned bank of realistic data-engineering tasks that grows continually; part of every bank is held out so the agents are never tuned against it.
• •Deterministic grading first. Pass/fail checks against fixtures and a full record of every run, as set out on our methods page. A new eval case becomes ground truth only after an independent review.
• •Self-improving skills and agents. We measure a skill by running the same agent with and without it on the same tasks. A proposed change ships only when it beats the version it replaces, and it still goes through code review.
• •Corrections land in the context, once. Agents work against the Data Context Wizard, a governed graph of schema, ownership, metrics, PII tags and lineage. Agents write back what they learn, and the places they get stuck become a ranked list of context to add.
• •The human placed by consequence. Every write is proposed first, a named person approves anything irreversible, autonomy is set per domain, and every change carries a signed receipt and can be reversed in one click.
• •Your stack, your eval set. In a pilot we build the eval set from your own recurring questions and past incidents, with expected answers frozen from your own systems, and re-run it at each milestone.

What we are building next. Evals that keep running in production: every agent action traced and scored as it runs, failures you approve turned into a regression suite for your deployment, and skills that learn from your environment and stay private to you.

Read the technical paper

Our technical paper, Context Is Not All You Need, makes the case these write-ups point to: a governed context layer is necessary but not sufficient, because most wrong answers come from planning and implementation rather than from finding the wrong table. It describes the architecture we think follows - narrow specialist agents, evaluation in the path of every change, and human approval where the consequences are.

Read Context Is Not All You Need, or start with the benchmark view in the Data Agent Evals Report 2026.

Sources. OpenAI, "Inside OpenAI's in-house data agent" (29 Jan 2026); DoorDash, "Inside Vera, DoorDash's Data Agent" (16 Sep 2026); Hex, "How we built a lab to evaluate data agents" (22 May 2026); Uber, "Unlocking Financial Insights with Finch" (Jul 2025); Meta Engineering, "Creating AI agent solutions for warehouse data access and security" (13 Aug 2025), with recall figures via PPC Land (17 Aug 2025); Analytics at Meta, "Inside Meta's Home Grown AI Analytics Agent" (30 Mar 2026); Anthropic, "How Anthropic enables self-service data analytics with Claude" (3 Jun 2026); Notion, "How we built a personal data scientist for every Notion employee" (8 Oct 2026); Red Hat, "We built an enterprise data agent, and you can too" (20 Aug 2026).

On the figures. Every number is the publishing team's own, about its own system, checked against its post on 19 October 2026. Notion's product page gives different usage figures (60% of GTM, Product and Ops staff daily; 5x fewer requests); we use its engineering blog's. If we have something wrong, email us and we will correct it and say so.