Data Agent Evals Report 2026

A year ago the question was whether an AI agent could write a SQL query. In 2026 the question is whether you can trust one to build a dbt model, answer a revenue question for a VP, or approve a request for access to a sensitive table - and how you would know. That is an evaluation question, and in 2026 the teams building these agents started publishing how they answer it.

We read all of it: the new benchmarks from Snowflake, Tencent and Databricks, the production write-ups from OpenAI, DoorDash, Notion, Anthropic, Meta, Uber, Hex and Red Hat, and OpenAI's own argument for why businesses need evals of their own. Five findings come up again and again.

• •Reliability is the number that matters, and it is much lower than best-case accuracy. Every pairing on data-eng-bench loses between 8.8 and 15.8 points when it has to solve a task three times out of three.
• •The harness is as much a variable as the model. The same model, on the same tasks, cost nearly four times as much in one harness as in another.
• •LLM judges flatter data agents. On identical outputs, two LLM judges scored the process more than 50 points higher than rule-based grading.
• •The eval set is mined from real failures, not written from scratch. DoorDash, Anthropic and Notion all turn production corrections into eval cases or context fixes.
• •Humans sit at the points of consequence, not on every answer. Plan approval, access decisions and context ownership are where the production teams put a person.

The benchmarks moved from SQL to pipelines

The text-to-SQL benchmarks that defined the first wave ask an agent to answer one question with one query. The 2026 benchmarks ask it to do data engineering: build models inside a real project, repair broken pipelines, and leave the warehouse in a state that passes tests it cannot see.

data-eng-bench (Snowflake AI Research, August 2026) is the most demanding public example. It has 103 tasks - 84 that build new models and 19 that fix broken ones - over 579 source tables in 19 schemas with roughly 8,000 columns. Each task is checked by a hidden verifier of 10 to 50 assertions, and a task only counts as solved if every assertion passes. The gold solutions look like real dbt work: 82% use ref, they average about nine ref calls, and the largest touches a DAG of 42 models.

It sits beside ADE-bench from dbt Labs (65 tasks as of August, by Snowflake's count), ELT-Bench (100 pipelines, 835 source tables, 203 target models) and DataClawEval from Tencent, which spreads 100 tasks across PySpark, MySQL, HiveSQL, Presto and FlinkSQL. The common shape is the point: a sandboxed project, a task written the way an engineer would receive it, and a deterministic check of the result.

Why is Pass^3 the number to watch?

Most leaderboards report Pass@1: how often an agent solves a task in a single attempt, averaged over runs. data-eng-bench also reports Pass^3: the share of tasks the agent solves in all three of three independent trials. For a team deciding whether to let an agent touch production, the second number is the one that matters, because an agent that is right two times in three is an agent somebody has to check every time.

The chart at the top of this report shows the gap for every pairing. It is widest for the smaller model. Sonnet 5 solves 56.6% of tasks on a single attempt in both harnesses, but only 40.8% every time - a 15.8-point drop. Opus 5 loses under 10 points. A better model buys consistency at least as much as it buys capability, and a single-run score hides exactly that.

Does the harness matter as much as the model?

The finding that surprised us most is how much of the result belongs to the harness: the loop, tools and context management wrapped around the model. Snowflake's own summary is that Cortex Code scored 4 points higher Pass@1 than Claude Code with Opus 5 at 3.9x lower cost, taking 2.2x fewer steps, 1.7x fewer SQL queries and 1.5x fewer tool operations.

Databricks found the same thing on its own multi-million-line codebase. "The harness a model is called from dramatically impacts cost and quality," its team wrote; the same model's cost differed by more than 2x across harnesses in some cases. It also found that cheaper tokens do not mean a cheaper task. Sonnet 5 is about 1.7x cheaper per token than Opus 4.8, but used 1.9x more tokens, so it cost $2.09 a task against $1.94 - while solving 81% of tasks against 87%.

The practical consequence: an eval that fixes the harness and varies only the model is measuring half the system. If you are choosing a data agent, evaluate the whole thing - model, harness, tools and context - on your tasks, and report cost per solved task, not cost per token.

Can an LLM judge grade a data agent?

Using a model to grade a model is the cheapest way to score open-ended work, and for data work it appears to be the least trustworthy. DataClawEval measured it directly: it graded the same Claude Opus 4.8 outputs with its rule-based protocol and with two LLM judges, GLM 5.2 and DeepSeek V4 Pro.

The paper's conclusion is blunt: "We therefore adopt rule-based grading as the definitive evaluation protocol for our benchmark." Databricks reached the same view independently: "We did not use an LLM judge to evaluate correctness, since we've found that this rewards sounding right over being right."

That does not make judges useless. DoorDash uses an LLM judge on its Vera agent to score tool usage, table selection and answer correctness, and OpenAI's grader compares the agent's SQL and its result data against a golden answer. The pattern that holds up is a judge anchored to something checkable - a golden query, a result set, a schema - rather than a judge asked whether an answer sounds right.

DataClawEval is also a reminder that "data engineering" is not one skill. Scores varied far more by engine than by model: PySpark results ranged from 48.0 to 83.8 across the models tested, FlinkSQL from 44.9 to 85.0, and no model passed 69.8 on HiveSQL. An agent that is strong on your MySQL estate tells you little about your Hive one.

How production teams build their eval sets

The public benchmarks say how capable the models are. The production write-ups say how teams decide an agent is good enough for their own data, and they agree on more than you might expect.

• •OpenAI evaluates its in-house data agent, which spans over 600 petabytes across 70,000 datasets, against golden question-and-answer pairs, each with a hand-written golden SQL query. The checks run continuously during development "like unit tests" and as canaries in production.
• •DoorDash's Vera runs an eval suite of more than 1,900 questions across 19 business domains. New cases are mined: a frontier model reads Slack threads and Jira tickets and drafts eval proposals, and domain owners accept about 95% of the de-duplicated candidates.
• •Hex built Shoebox, an eval lab that runs every change as a pairwise experiment - a candidate against a baseline - inside Shorelane Commerce, a synthetic business with six years of data where five different columns could plausibly be called "revenue".
• •Anthropic took its internal analytics agent from 21% to consistently above 95% in aggregate on its evals by giving it skills: domain reference documents behind a router, and a runbook encoding the process a senior analyst would follow.

OpenAI's guidance for businesses frames all of this as product management: evals "make fuzzy goals and abstract ideas specific and explicit," and its own frontier evals cannot capture any one company's workflows, so each company needs its own. Its summary line is the best one-sentence version of this report: "Don't hope for 'great.' Specify it, measure it, and improve toward it."

We cover each of these systems, and four more, in detail in our companion piece: How Large Companies Evaluate Data Agents in Production.

Where does the human belong in the loop?

No production team in this set puts a person on every answer, and none removes people entirely. They put the person where an error would be expensive or where the agent cannot know the truth.

• •Before the expensive step. DoorDash's /analysis workflow writes an explicit plan - timeframes, metrics, scope, constraints - and nothing executes until the user approves or adjusts it.
• •On the access decision. Meta's warehouse agents triage and draft access requests and help data owners work through their security procedures, and the agent was evaluated against the decisions people had already made.
• •On the context, not the query. Anthropic and Notion turn corrections into fixes to shared context, and Red Hat's agent drafts guidance from query logs for data product owners to review, so a fix lands once rather than in one conversation.
• •Under the user's own permissions. OpenAI's agent is pass-through: "users can only query tables they already have permission to access." Notion's Data Scout has each user sign in to Snowflake with their own credentials.

What we would do with this

If you are building or buying a data agent, this is the evaluation plan we would copy from the teams above:

• •Score reliability. Run each task at least three times and report the share solved every time, next to the single-run number.
• •Grade against something checkable. Golden queries, result sets and hidden assertions first; an LLM judge only where it is anchored to one of those.
• •Evaluate the whole system, on your data. Model, harness, tools and context together, on tasks drawn from your own warehouse and engines, with cost per solved task.
• •Mine the eval set from production. Every correction a person makes is a candidate eval case; have the domain owner approve it.
• •Put people at the points of consequence. Plan approval before expensive work, owners on access and on definitions, and the user's own permissions everywhere.

How Data Workers evaluates its own agents

We build the Autonomous Agentic Data Platform, so the practices above are ones we hold our own agents to. Here is how they show up in the product.

• •Held-out task banks. Each of our 20 agents is scored against versioned banks of realistic data-engineering tasks that grow continually, and part of every bank is held out so the agents are never tuned against it.
• •Grading you can check. Deterministic pass/fail checks, held-out fixtures and a full record of every run. We quote no score without its run parameters; the protocol is on our methods page.
• •Reviewed ground truth. A new eval case becomes ground truth only after an independent review, never on the word of the process that generated it.
• •Self-improving skills and agents. We measure a skill by running the same agent with and without it on the same tasks, and cut or rewrite skills that do not help. A proposed change to a skill or agent ships only when it beats the version it replaces, and it still goes through code review.
• •A context layer that learns. Agents work against the Data Context Wizard, a governed graph of schema, ownership, metrics, PII tags and lineage in which every fact carries its source. Agents write back what they learn, and the places they get stuck become a ranked list of context to add.
• •Humans at the points of consequence. Every write is proposed first, a named person approves anything irreversible, autonomy is set per domain, and every change carries a signed receipt and can be reversed in one click.
• •Tuned to your stack. In a pilot we build an eval set from your team's own recurring questions and past incidents, freeze the expected answers from your own systems, and re-run it at each milestone to tune the deployment to your warehouse, engines and conventions.

What we are building next. Evals that keep running in production: every agent action traced and scored as it runs, failures you approve turned into a regression suite for your deployment, and skills that learn from your environment and stay private to you.

Read the technical paper

These findings line up with the argument we make in our technical paper, Context Is Not All You Need: a governed context layer is necessary for a data agent, but it is not sufficient. Most wrong answers come from planning and implementation, not from failing to find the right table, and once an agent can write to your warehouse, a planning error becomes an incident. The paper sets out the architecture we think follows - narrow specialist agents, evaluation built into the path a change travels, and human approval placed where the consequences are.

Read Context Is Not All You Need, and if you are running evals on data agents of your own, we would like to compare notes.

Sources. Snowflake AI Research, "Introducing Data-eng-bench" (6 Aug 2026); Tencent, DataClawEval, arXiv 2607.28033 (30 Jul 2026); Databricks, "Benchmarking Coding Agents on Databricks' Multi-Million Line Codebase" (8 Jul 2026); dbt Labs, ADE-bench; Jin, Zhu and Kang, ELT-Bench, arXiv 2504.04808; OpenAI, "Inside OpenAI's in-house data agent" (29 Jan 2026) and "How evals drive the next chapter in AI for businesses" (19 Nov 2025); DoorDash, "Inside Vera, DoorDash's Data Agent" (16 Sep 2026); Hex, "How we built a lab to evaluate data agents" (22 May 2026); Anthropic, "How Anthropic enables self-service data analytics with Claude" (3 Jun 2026); Meta Engineering, "Creating AI agent solutions for warehouse data access and security" (13 Aug 2025); Notion, "How we built a personal data scientist for every Notion employee" (8 Oct 2026); Red Hat, "We built an enterprise data agent, and you can too" (20 Aug 2026).

On the figures. Every benchmark result here is the publisher's own, on its own benchmark, and several publishers ship a product they test. We report their numbers as published and do not rank vendors from them. Each was checked against its source on 19 October 2026, and figures available only second-hand are marked as such. Data Workers does not appear in any of these benchmarks. If we have something wrong, email us and we will correct it and say so.