Metric Disambiguation by Hand vs With an AI Agent

Metric disambiguation is the job of deciding which of several metrics that share a name (for example, three different "revenue" numbers) actually answers the question someone asked. Done by hand, it means reading SQL, dbt models, and dashboards and then asking the metric owner. An agent can do the lookup and comparison in one pass over the catalog, but a person should still approve the choice whenever the number will leave the data team.

Key Takeaways

• •Metrics that share a name usually differ in one of three things: grain, filter, or source. Disambiguation means finding which of the three differs.
• •By hand, the work is mostly archaeology: reading model SQL, comparing dashboard definitions, and messaging owners. The slow part is waiting on people, not reading code.
• •An agent can compare the candidate definitions side by side and recommend the one that fits the question, as long as it can read your catalog and transformation code.
• •A human should approve any choice that feeds an executive report, a customer-facing number, or a change to the canonical definition.
• •Evaluate an agent on questions you have already resolved by hand. Its picks should match yours, and it should say when it cannot decide.

Metric disambiguation by hand vs with an AI agent

We wrote this for data leads who keep getting the same Slack question: "Which revenue number is the right one?" It is a pillar topic for us because it is the clearest case where a coding agent working against your catalog does something a dashboard cannot. We try to be honest about where doing it by hand is still the better call.

The agent we describe here is the Catalog Agent, one of the fifteen MCP-native agents in Data Workers. It carries a skill called dw-metric-disambiguation. That skill's job, as we describe it: disambiguate metrics that share a name but differ in grain, filter, or source, and pick the one that fits the question. We limit our claims to that.

The job and who does it today

In most companies nobody owns this job. An analyst hits it first, when two dashboards disagree. An analytics engineer hits it second, when a stakeholder asks why finance and sales report different bookings. A data lead hits it third, usually when the disagreement shows up in a board deck.

Conflicting definitions are the reason semantic layers exist. The dbt Semantic Layer documentation describes defining a metric once and reusing it across tools, and the MetricFlow repository holds the open-source engine behind that idea. If you already have a clean semantic layer, disambiguation is a smaller problem. Most warehouses we see described by our users have a semantic layer covering some metrics and a long tail of legacy ones that it does not.

The manual workflow, step by step

• •Find the candidates. Search the BI tool, the catalog, and the dbt project for the metric name. Expect different spellings: revenue, net_revenue, rev_usd, total_sales.
• •Read each definition. Open the model SQL or the dashboard query. Note the grain (per order, per customer-day, per month), the filters (refunds excluded? test accounts excluded? status values?), and the source tables.
• •Compare them. Build a short table of differences. This is the step that tends to live in someone's head or a scratch doc.
• •Check usage. Find out which definition feeds which dashboards, and who looks at them. A definition nobody queries is a weaker candidate.
• •Ask the owner. Message the person named in the model's meta, or whoever last touched the file. This is where the process stalls, because the owner may have left or may not remember.
• •Map the question to a definition. Decide which candidate fits what was actually asked, for example "revenue recognized this quarter" versus "revenue booked this quarter".
• •Write it down. Update the docs or the catalog so the next person does not repeat the work. In our experience this step is the one most often skipped.

Where the manual workflow breaks

Three failure modes show up repeatedly. First, the candidate search misses a definition because it uses a different name. Second, two definitions look identical in SQL but differ in an upstream model's filter, so the difference only appears three models back. Third, the answer lives in one person's memory and is never recorded, so the same question returns next quarter. None of these are skill problems. They are the cost of reading many files and tracking many hand-offs.

What an agent does instead

Working from the skill description, the agent handles the middle of that workflow: it takes a question such as "what was revenue last quarter?", finds metrics that share the name, separates them by grain, filter, and source, and recommends the one that fits the question. Because the Catalog Agent) federates across catalogs such as OpenMetadata, DataHub, Amundsen, Unity Catalog, and Atlan, the candidate search is not limited to one tool's view.

Because each of our agents is an MCP server, you can invoke this from Claude Code or Cursor without opening a new dashboard. The Model Context Protocol is the open standard that makes that possible. A prompt like "Which revenue metric should I use for the Q3 board deck?" runs where you are already working. We covered how our agents chain together in other posts; for this job the Catalog Agent can hand lineage questions, such as what feeds a candidate, to the Schema Agent.

What still needs a human

• •Approving the choice for anything that leaves the data team: board decks, investor updates, customer-facing numbers.
• •Declaring a canonical definition. An agent can recommend one; the business owns what the company means by revenue.
• •Resolving genuine disagreement. If finance and sales define bookings differently on purpose, that is a policy decision, not a lookup.
• •Retiring or renaming metrics that other teams depend on.

The open-source edition of Data Workers (dw-claw) is read-only by default, which fits this job well: disambiguation needs reading, not writing. Writing the decision back to a catalog, with approval workflows and an audit trail, is a Pro feature.

Manual vs agent: side-by-side

Manual still wins in two cases. When there are only two candidates and you wrote one of them, you will be faster than any tool. And when the real issue is a policy dispute, an agent will not help you settle it.

How to evaluate an agent for this job

We would run the same test whether the agent is ours or anyone else's. Take five to ten metric questions your team has already resolved by hand, and for which you know the right answer. Ask the agent each one and score three things.

• •Accuracy: did it pick the definition your team picked?
• •Evidence: does it show the grain, filter, and source differences that led to its pick, so you can verify them in under a minute?
• •Honesty: when two candidates both plausibly fit, does it say so and ask, instead of guessing confidently?

Also check access scope. The agent should read your catalog and code without needing write permissions, and your PII and audit requirements should hold wherever it runs. We keep data in the customer's infrastructure for this reason. We have no public benchmark for this skill and will not quote one; run the test on your own metrics.

Frequently Asked Questions

What is metric disambiguation? It is choosing the right metric when several share a name but differ in grain, filter, or source, so the number you report matches the question being asked.

Do I need a semantic layer before using an agent for this? No. A semantic layer such as the one in dbt reduces the number of conflicts, but the agent is most useful exactly where definitions are scattered and undocumented.

Can an AI agent decide which metric is canonical? It can recommend, with evidence. The canonical definition is a business decision, so a named owner should approve it.

Does this work from Claude Code or Cursor? Yes. Each Data Workers agent is an MCP server, so any MCP-compatible client, including Claude Code, Cursor, Windsurf, and VS Code, can call it.

Is the open-source version enough? For disambiguation, usually yes, since the work is read-only. Writing the outcome back to a catalog with approvals and audit trails needs Pro.