Benchmark methods: how to evaluate an agentic data engineering agent

1. Why a protocol before results

A benchmark number published before its method is unfalsifiable, and a method published after the number is a rationalisation of it. Publishing the protocol first fixes the order: the rules are on the record, dated, before anyone knows who they favour.

The category needs it. Every vendor in agentic data engineering now uses the word agent for six different behaviours, from detecting a condition and notifying a human, through drafting a change for a person to apply, to applying a change unattended and then verifying the outcome. That is the six-level scale used in the Agentic Data Engineering Index, which graded 21 vendors including Data Workers on evidence from public pages, one cell at a time, on 9 September 2026. An index grades what a vendor evidences. A benchmark grades what an agent does. This page is the second thing, and it does not exist yet as a run.

The other reason is that the alternative failed in public. Benchmarks for data agents have been gamed, and the mechanisms are documented rather than alleged, which section 6 sets out with sources.

2. The task set

Six categories, taken from the six lanes of the Agentic Data Engineering Index Q4 2026 so that a benchmark result and an index cell describe the same axis. Each is a fixture the runner can rebuild from a commit, not a scenario a human interprets.

A category the product under test does not address is recorded as not evidenced here, not as a zero. Grading an absent capability as a failure inflates the spread between products that do different jobs, which is the same rule the Index applies to single-platform products in the cross-cloud lane.

3. Scoring is deterministic

LLM-as-judge scoring is not credible for this category, and the reason is specific rather than general suspicion of model graders. The ground truth here is executable, so a judge is a strictly worse instrument than the thing it replaces: where a script can settle whether the build succeeded and the counts reconcile, a judge introduces variance in exchange for nothing. It is also correlated with the system it grades, because the agent under test and the judge are often the same model family, and a grader that shares the failure modes of the subject will forgive them. It cannot be recomputed by a reader, so the run record stops being checkable and becomes a transcript of an opinion. And it drifts: upgrade the judge and last quarter’s numbers move, which means the benchmark was measuring the judge. Prose quality is the one thing a script cannot settle, so this protocol does not grade prose. In the catalog category the machine-checkable fields are graded, lineage edges, owner assignments, semantic labels on a closed vocabulary, and the descriptive writing is left ungraded rather than handed to a judge.

The cost of this rule is honest: deterministic checks cannot measure whether an agent’s explanation was any good, and they reward a task set that is narrower than the job. We would rather publish a narrow number a reader can recompute than a broad one they have to trust.

4. What a valid run record contains

Seven fields. All seven, or the number is not a result.

1. 1.Fixture commit SHA. The exact commit of the fixture repository and of the warehouse seed the run executed against. A named version is not enough, a run against a fixture nobody can rebuild is not reproducible.
2. 2.Model and version. The model identifier, the provider, and the dated version string. A family name such as the vendor plus a generation is not a version.
3. 3.Exact agent configuration. Tools available, system prompt, retry policy, temperature or its absence, the autonomy setting, and any scaffolding around the model. The whole configuration, in the form it was run.
4. 4.Wall-clock time and token cost. Per task and in aggregate. An agent that passes a check after nine hours and forty dollars has told you something different from one that passes in four minutes.
5. 5.The full transcript. Every trial, including the failures, including the runs excluded from the aggregate and the reason each was excluded. Publishing only the passing trials is the oldest way to move a number.
6. 6.Pass or fail per check. Not a headline percentage. Each deterministic assertion, per task, per trial, so a reader can recompute the aggregate from the record and get the same number.
7. 7.Who ran it. A named person, and whether they are affiliated with the vendor whose agent was tested. Self-run is allowed. Anonymous is not.

5. Contamination rules

The fixture must be new, or held out. A public benchmark the model may have trained on is not evidence, because a passing trial cannot be separated into work the agent did and an answer it remembered.

• •New or held-out fixtures only. A fixture is valid if it was created for the run, or if it was withheld from public release. A public benchmark that the model may have trained on is not evidence about the agent, because you cannot separate retrieval of a memorised answer from work.
• •No gold at agent time. Expectation files, answer keys, reference solutions and grading tests are absent from the environment while the agent runs, and are copied in at grading time only. This is proven per trial by a shield check, not asserted by the runner.
• •No gold in the selection either. Trials are independent end-to-end runs. Choosing which trials to submit by consulting the answer key turns a per-trial metric into a best-of-N metric while it still wears the per-trial name.
• •Clean environment attestation. Runs execute in a fresh container with no warm dataset caches. Local caches and benchmark checkouts outside the sandbox root are swept before the run. A network sandbox with a warm local cache is not a sandbox.
• •Access manifest per trial. The per-trial connection and file manifest ships in the evidence bundle: structural proof of what the agent could reach. An assurance that no answer-key path existed is not that proof.
• •Memorisation caveat where it applies. Where a fixture derives from a public labelled corpus, the record says so in the same breath as the number. A sandboxed model recalling public labels is not agent capability, and no substring detector catches the silent case.
• •Fail closed. The contamination audit runs between execution and scoring for every task. A trial whose audit errors is rejected exactly as a tainted trial is. An audit that fails quietly passes tainted work.

One consequence is worth stating on its own, because it is the rule people find inconvenient. Carrying an artifact produced during one task into another task of the same fixture project is contamination, even when the carry is governed and logged, because a passing earlier trial ends in a state the grader has certified as correct. That is an answer key with a different means of transport. Data Workers ruled on exactly this question against its own interest on 9 September 2026, in an internal integrity ruling covering our own benchmark runs. That ruling is a statement about how we run, not evidence about anyone else.

6. Where these rules came from

None of the rules above were invented for this page. They are the eight items of an internal benchmark publish-integrity checklist, first committed to the Data Workers agent swarm repository in June 2026, and they were written in response to mechanisms observed on a public leaderboard rather than in the abstract.

The leaderboard was DataAgentBench, the data-agent benchmark maintained by the UC Berkeley EPIC lab at github.com/ucbepic/DataAgentBench. Two mechanisms visible there in June 2026 are the direct reason for two rules on this page. We describe each mechanism below. Where the mechanism can be read from a public artifact, that artifact is linked and we fetched it on September 10, 2026 to confirm it resolves. Where it cannot, we keep the mechanism and drop the name, because the rule is justified by how the mechanism works and not by whose submission it appeared in.

Trial selection that consults the answer key. The SCRIBE submission from Actioneer, at DataAgentBench pull request 67, assembles its five trials per query from a pool of many runs. The assembly script is public in the submitter’s own repository at scribe-dab/scripts/build_dab_submission_best5.py, and its own docstring describes the method: from every available run directory, validate each candidate, sort passing answers first, take the first five, and pad with failing answers as needed. The validation step executes the benchmark’s official gold-checking script. Both links were fetched on September 10, 2026. This is disclosed openly by the submitter rather than concealed, so it is a comparability question and not an accusation, and it is exactly why this protocol requires trials to be independent end-to-end runs whose selection never consults gold.

A recompute that re-scores superseded submissions. The second mechanism, which we are deliberately not attaching to a vendor name: a leaderboard-wide recompute rebuilt rows from the submission files held on the pull-request branches, and those files still contained trials that the maintainers had already adjudicated as leaked, while the corrected replacements existed only as attachments on comments and were never pushed to the branch. We are not naming the rows here, because establishing that claim requires comparing several artifacts and a reader cannot check it from any single URL we could give them. The mechanism is what matters, and it is the reason this protocol pins the validator version and requires the full transcript, including every excluded trial and the reason it was excluded.

Naming one submission and not the other is a sourcing rule, not a judgement about either team. The rule this page applies to itself is the one it asks of everyone: a statement about somebody else needs a public artifact a reader can open, and where there is none, the mechanism goes in the record and the name stays out of it.

7. Applying this to ourselves

We are stating that here, on the protocol page, rather than in a footnote somewhere quieter, because a company that publishes rules for benchmark integrity and exempts its own numbers from them has published marketing. The same discipline is already applied in the Agentic Data Engineering Index, where Data Workers is graded from its public site and public GitHub only, is not graded at the top autonomy level in any lane, and where the site’s benchmark figure is deliberately not cited because no public run record accompanies it.

8. How to run it against us, or anyone

The protocol text on this page is released under the Creative Commons Attribution 4.0 licence. Anyone may run it, against any vendor, including us, including a competitor, and publish the result without asking. Cite it as: Data Workers, Benchmark methods for agentic data engineering, 10 September 2026.

What you need from us to run it against Data Workers is public already. The open-source core is Apache-2.0, eleven agents and more than 160 MCP tools, read, analyse and recommend only, at github.com/DataWorkersProject/dataworkers-claw-community. It runs in Claude Code, Cursor, Codex CLI, OpenCode, Gemini and any MCP client. If you need a platform deployment configured for a run, ask and we will set one up rather than send a deck.

To contest a result, ours or a competitor’s, send the run record to hello@dataworkers.io. A record with all seven fields moves a claim on this site. A statement without one does not, which is the same rule the Index applies to contested cells, reviewed within ten business days.

Pricing, so it is on the same page as the protocol: the open-source core is free, Pilot Program $7,500 one-time, then Scale from $1,000 per month, no usage meter, as published on 10 September 2026. Enterprise starts at $3,000 per month.

9. When this protocol is the wrong tool

It is the wrong tool if what you are buying is a dashboard or a chat-to-SQL analyst surface. The six categories test repair, review, build and documentation of pipelines, not analytical question answering, and a product that is good at the second will score badly here for reasons that do not matter to you.

It is the wrong tool if your decision turns on procurement rather than capability. No benchmark number answers a security or procurement requirement. We walk through those with you on a call, separately from any run.

And it is the wrong tool if you want a single ranking. Six categories with pass and fail per check do not collapse into one number without a weighting, the weighting is a judgement about your estate rather than a fact about the agents, and any vendor who hands you the collapsed number has made that judgement for you.

10. Common questions