Agents proven on benchmarks and on your own data
A benchmark score tells you how an agent does on someone else's tables. Data Workers is evaluated on DataAgentBench with a published method, then builds an eval set from your own historical questions during onboarding and keeps it as a regression suite.
Before and after
| Today | With Data Workers |
|---|---|
| Vendor scores you cannot reproduce | A published method on a public benchmark, with uncertainty reported |
| A demo on clean sample data | An eval set built from your own historical questions |
| Expected answers that nobody agreed | Expected answers confirmed by your data owners |
| A bake-off before you buy, then nothing | The same eval kept as a regression suite after go-live |
| "It seemed fine" after a model or schema change | Every change re-scored against the same questions |
A public benchmark shows that a method works; it cannot show that an agent knows your revenue definition or which of your three customer tables to trust. Models, prompts and schemas change every month, so a score taken once at purchase goes stale. The 2026 evals report and how large companies evaluate data agents in production both land on the same practice: test on your own questions, and keep testing.
How it works
1. Turn your query history into an eval set
During onboarding the Data Context & Catalog Agent reads your query history with analyze_query_history and surfaces the questions your team really asks, in your words and against your tables. Your data owners choose which ones belong in the set. Nothing is invented for the test.
2. Agree the expected answers
For each question, the owner confirms the right answer against the source your catalog marks as authoritative, using get_authoritative_source. Those answers become the fixed reference, so a pass or a fail means the same thing next month as it does today.
3. Score every change and keep watching
The Observability Agent reports results with get_evaluation_report and flags movement with detect_drift. When the model, a prompt, a schema or a definition changes, the same questions run again and any answer that got worse goes to the domain owner. For single-answer trust (sources, refusing to guess), see data agents you can check, not just trust.
The impact metrics we move
- •An eval set from your own questions: from about 3 weeks to 2 days
- •Re-running the full eval after a change: from about a week to ~2 h
- •Finding which questions got worse: from about 2 days to under 30 min
Where teams start
- •Before you buy. Read the benchmark methods page: how Data Workers is evaluated on DataAgentBench, how uncertainty is reported, and why no lead is claimed.
- •One domain's most-asked questions. Finance or sales questions your analysts answer weekly, with a named owner for the expected answers.
- •Before a model or prompt change. Run the eval first, then decide whether to switch.
- •Before raising autonomy. Give the domain owner evidence beyond a demo when they decide whether agents may do more.
- •Text-to-SQL you already run. Score it on the same questions; what makes text-to-SQL reliable is a useful checklist.
What powers it
The Data-Agents Swarm does the work: the Data Context & Catalog Agent supplies your real questions and the authoritative source for each answer, and the Observability Agent reports results and watches drift. The Spellbook Data Catalog holds the definitions the expected answers rest on, the Data Context Wizard keeps that context current, and Autopilot Routines re-run the eval on the schedule you set. The Autonomous Data-Conductor reads the results when you decide how much a domain may do on its own. For the wider field, data agent evaluation frameworks compares approaches.
Where this shows up: data quality and governance, enterprise AI context and AI and ML platform data teams.
The climb: from assisted to autonomous to agentic
Every transformation runs on the same ladder from our thesis: agents first watch and suggest with receipts, then act after approval, and a domain moves up only on its own review record. L0 to L2 run today. L3 and L4 are where a domain can go on evidence, and an eval set built from your own questions is part of that evidence.
The pilot
- •Scope. One data domain and the questions its team asks most.
- •First week. Agents start by watching and suggesting, read-only. We build the eval set from your query history and show the agent's answers beside the expected ones.
- •By week two. Owners have confirmed the expected answers, and the eval re-runs after changes on its own; anything sensitive stays with a named approver.
- •At day 30. We review the impact metrics against your baseline:
- •An eval set from your own questions: from about 3 weeks to 2 days
- •Re-running the full eval after a change: from about a week to ~2 h
- •Finding which questions got worse: from about 2 days to under 30 min
- •Cost. Start with a pilot ($7,500 one-time, credited in full against the first year). Scale is from $1,000 a month and Enterprise from $3,000 a month, billed annually, with unlimited seats, no usage meter and no markup on model spend. See pricing.
FAQ
Do you publish a benchmark score? Not on this page. We are evaluated on DataAgentBench, a public benchmark of real multi-database analytics questions, and the methods page explains how we measure and report uncertainty. We claim no lead on any leaderboard.
How is this different from Spider or BIRD? Those benchmarks test SQL generation against a database. DataAgentBench asks questions that span several databases, and your own eval set tests what neither can: your definitions and your tables.
Who decides what the right answer is? Your data owners. Agents propose questions and point to the authoritative source; a person confirms every expected answer.
What if the agent gets one wrong? When the data to answer is not there, it says so instead of guessing. When a person corrects an answer, the correction is remembered, and a question that gets worse after a change goes to its owner.
Does the eval data leave our environment? Not if you run Data Workers in your own environment; your questions and answers stay on your side.