Agent As A Judge Data

Agent-as-a-judge is an evaluation pattern where one AI agent grades the output of another, using structured rubrics and tool access instead of static test cases. Applied to data workflows, it means using a scoring agent that can inspect schemas, run sample queries, and verify lineage before approving a sister agent's plan or code.

The pattern emerged in early 2026 as a response to the unit-test ceiling: you cannot write static asserts for every SQL query an agent might generate, but you can teach a second agent to judge whether the result is reasonable. This guide explains the pattern, where it beats classic evals, and how it plugs into data infrastructure.

Agent-as-a-Judge vs Traditional Evals

Traditional LLM evals compare generated text to a reference string or run a fixed assertion. That works for summarization or Q&A, but breaks on data tasks where many queries are semantically correct. Agent-as-a-judge replaces brittle string comparison with a live judgement: the judge runs the output against a sandbox, inspects the result, and grades it against a rubric.

The practical gap shows up on day one. A pipeline agent generates a dbt model that computes revenue a different way than the reference but is actually correct because it accounts for refunds the reference missed. A string-matching eval fails it. A judge agent runs both queries, compares the outputs, checks the business definition of revenue in the catalog, and scores the new version higher. That kind of semantic evaluation is impossible with static test cases.

Why It Matters for Data

Data agents generate SQL, Python, dbt models, and terraform. Each of these has many valid forms. A strict reference-matching eval would reject perfectly good output. A judge agent can run the SQL, compare row counts, check column types, and verify nothing hit a PII column — graded holistically, not token-by-token.

• •Semantic correctness — does the query return the right answer?
• •Schema safety — are all referenced columns real?
• •Policy compliance — no PII, no forbidden joins?
• •Performance — did the query finish under the budget?
• •Readability — is the code maintainable by humans?
• •Idempotency — can the operation be safely retried?

How to Build a Data Judge Agent

The core ingredients are a rubric, tool access, and a sandbox. The rubric defines what 'good' means (typically five to eight criteria). Tool access lets the judge inspect catalog, run queries, and read policies. The sandbox prevents the judge from touching production. A judge without tool access is just a second model guessing; a judge with tools is a meaningful evaluator.

Start with a minimal judge that has three capabilities: run the generated SQL against a staging warehouse, compare the result schema to the catalog, and check the query plan for full table scans or cross-joins. Those three checks catch the majority of data agent failures and the implementation is straightforward. Add policy checks, cost estimation, and readability scoring in later iterations once the core loop is stable.

Rubric Design for Data Tasks

A good data-task rubric has six to eight criteria, each scored on a one-to-five scale with concrete anchors. The anchors matter more than the criteria themselves. 'Correct SQL' is ambiguous; 'query returns row counts within one percent of the ground truth and uses only columns that exist in the catalog' is a real anchor. Teams that ship rubrics without anchors see judge scores drift week over week because each run of the judge interprets the criteria slightly differently. Anchors pin the interpretation and make the scores comparable across runs, agents, and model versions.

The rubric should also distinguish between hard failures and soft penalties. A hard failure is a query that touches a forbidden PII column or drops a production table. A soft penalty is a query that returns the right answer but is hard to read. Mixing the two produces a scalar score that hides the difference between 'unsafe' and 'ugly' — and unsafe should never ship while ugly can. Split the rubric into safety criteria (binary pass or fail) and quality criteria (graded), and route the two categories to different escalation paths.

Agent-as-a-Judge in Data Workers

Data Workers ships a dedicated evaluation agent that grades the output of every pipeline, catalog, and governance agent before promotion. It runs the SQL against a staging warehouse, compares row counts to historical baselines, checks lineage completeness, and enforces PII policies — all scored on a 1-5 rubric. Failures block promotion and log a trace for human review. See AI for data infrastructure for how this fits the full agent swarm, or read our notes on DataAgentBench for the benchmark that grew out of this work.

Operating the Judge at Scale

Running a judge on every agent output is expensive. Practical deployments sample at different rates per risk tier. CI runs every canonical scenario. Staging samples every run at 100 percent. Production samples low-risk runs at five percent, medium-risk runs at twenty percent, and high-risk runs at one hundred percent. The sampling is driven by the same risk tier the human-in-the-loop system uses, so a single policy drives both. Budget caps per team prevent runaway judge costs, and a fallback to a cheaper model kicks in when the primary judge is overloaded.

Latency is the second operational concern. If the judge takes thirty seconds to score a query, every pipeline run adds thirty seconds of wait time. The fix is async scoring for low-risk tasks — the pipeline proceeds while the judge scores in the background, and a retroactive alert fires if the score drops below threshold. High-risk tasks still block on the judge, but those represent less than ten percent of runs in most deployments.

Bootstrapping the First Judge

The first judge agent is always underwhelming because the rubric is immature and the ground truth is sparse. The practical path is to start with a rubric of three criteria, run it on twenty canonical scenarios, and let the scores be ugly on purpose. The ugly scores are the signal — they show where the rubric is ambiguous, where the ground truth is missing, and where the judge model disagrees with the humans. Fix those gaps, regenerate the scores, and repeat. After three or four iterations the judge converges on a stable rubric and the scores become meaningful.

Teams that try to ship a polished judge on day one never ship; teams that ship the ugly version and iterate ship within a month. The secret is treating the judge as a product with its own improvement cycle, not a one-shot setup script.

Common Mistakes

The most common failure is using the same model as both generator and judge without any external signal. The judge will rubber-stamp whatever the generator produced. Real agent-as-a-judge works when the judge has access to ground truth the generator did not: a different model, a live warehouse, or a test dataset. Another mistake is treating the rubric as final — rubrics must evolve as the team discovers new failure modes. A third mistake is scoring everything on a single scalar. Aggregate scores hide the distinction between safety failures and style issues, and that distinction is the whole point of structured evaluation.

Ready to see agent-as-a-judge applied to your pipelines? Book a demo and we will walk through the evaluation layer on live data.

Agent-as-a-judge replaces brittle test cases with live scoring by a second agent. For data workflows it is the only eval pattern that scales to generated SQL and pipeline code. Teams shipping production agents are already building judges; the rest are still writing static asserts.