How to stop AI agent using wrong metric definition: a 5-step tutorial
Pin revenue to one governed column so your agent stops guessing
To stop an AI agent from using the wrong metric definition, give it one governed definition to query instead of a warehouse full of candidate columns. Define each metric once in a semantic layer such as the dbt Semantic Layer, expose it to the agent over MCP, and instruct the agent to refuse raw-table answers when a governed metric exists. The rest of this post walks through those steps in order.
Key Takeaways
- •An agent picks the wrong revenue column because the warehouse offers several plausible ones and nothing tells it which one finance signed off on.
- •The fix is a single governed definition per metric, served through a semantic layer, not a longer prompt.
- •Restricting the agent's tools matters as much as defining the metric: if raw-table SQL is available, the agent will sometimes use it.
- •A small set of golden questions with known answers catches definition drift before a stakeholder does.
- •The failure is silent. The answer looks right, arrives fast and is wrong, so detection has to be built in rather than hoped for.
How to stop AI agent using wrong metric definition: why it happens
Ask a data agent "what was revenue last quarter?" and it will answer in seconds. The query it wrote may have summed gross order amount, net of refunds, recognized revenue or bookings, and nothing in the response tells you which. In our opinion this is the most dangerous failure mode in agentic analytics, because the output is fluent, confident and plausible. Nobody checks it until two dashboards disagree in a board meeting.
The cause is structural. A typical warehouse has columns like amount, gross_amount, net_amount, recognized_amount and revenue_usd spread across orders, invoices and finance marts. A text-to-SQL agent reads column names and descriptions, picks the closest match to the question and writes valid SQL. Valid SQL is not the same as the agreed definition. The agent is missing context, not intelligence: which column is canonical, which filters apply (test accounts, cancelled orders), and which currency and time grain the company reports in.
We call that missing piece the context layer. The steps below build a minimal version of it. We use Claude Code in the examples because that is where most of our readers run agents, but the approach is the same in Cursor or any MCP-compatible client.
Step 1: Inventory every candidate column for the metric
Before you define anything, find out how bad the ambiguity is. List every column that could plausibly answer the question "what is revenue?" Search column names for revenue, amount, total, price and sales across your marts and staging models, then note the table, the grain and who uses it.
You can do this by hand with information_schema queries, or ask the agent directly: "Which columns in analytics could be called revenue, and which tables use each one?" Our Catalog Agent does this across catalogs and the warehouse in one pass, but a spreadsheet works for a first run. The goal is a short list, usually between three and eight candidates, and a decision about which one is correct for each audience (finance, product, sales).
Expect an uncomfortable meeting at this step. If finance and product report different numbers, you have found a real definition conflict that no agent can resolve for you. Resolve it with people first, then encode the result.
Step 2: Define the metric once in a semantic layer
Write the agreed definition as code. In the dbt Semantic Layer, metrics are built on semantic models and measures through MetricFlow, as described in the MetricFlow docs. A governed revenue metric names the measure (for example, a sum over net_amount), the filters (exclude test accounts and cancelled orders), the time dimension and the owner. The point is that "revenue" now resolves to exactly one expression.
- •Name it unambiguously: net_recognized_revenue is better than revenue, because it tells the agent and the reader what it is.
- •Keep the old ambiguous names as deprecated aliases that point to the governed metric, so older questions still resolve.
- •Add a description written for a model reader: when to use this metric, when not to, and which other metric to use for bookings.
- •Record an owner and a review date so definitions do not rot.
If you do not run dbt, any semantic layer that exposes named metrics through an API serves the same purpose. What matters is that the definition lives outside the prompt and outside individual SQL files.
Step 3: Expose the metric to the agent over MCP, and take away raw SQL
A governed metric only helps if the agent actually calls it. Expose the semantic layer as an MCP server so the agent can list metrics, read their descriptions and query them by name. The Model Context Protocol is the open standard Claude Code, Cursor and other clients use to attach tools, so the same server works across them. Claude Code's setup for MCP servers is covered in the Claude Code docs.
Then narrow the tools. If the agent also has a general run_sql tool against the warehouse, it will use it whenever the metric lookup feels slower or less flexible. For analyst-facing agents, we recommend a read-only role that can reach marts only through the semantic layer, with raw-table access reserved for engineers. Our open-source edition is read-only by default for the same reason: an agent that cannot write or wander is easier to trust.
Our Insights Agent works this way. It produces text-to-SQL grounded by the semantic layer and returns the metric name, assumptions and citations with every answer, so a reader can see which definition was used.
Step 4: Write the rule into the agent's standing instructions
Tools constrain what the agent can do; instructions shape what it tries first. Put a short, explicit rule in the file your agent reads at the start of every session (CLAUDE.md for Claude Code, rules files in Cursor). A version that works in practice has four parts:
- •Always resolve business metrics through the semantic layer tools before writing SQL.
- •If no governed metric matches the question, say so and ask which definition to use. Do not pick a column.
- •State the metric name and definition in every numeric answer.
- •Never combine a governed metric with a raw-table filter that changes its meaning.
The second rule does the most work. An agent that is allowed to say "no governed metric matches" produces a visible gap that someone can fix. An agent that is not allowed to say it produces a silent wrong answer.
Step 5: Add golden questions and run them on every change
Definitions drift. Someone adds a new column, renames a measure or changes a filter, and last quarter's number moves. Protect against that with a regression set: 10 to 30 questions whose correct answers finance has already confirmed, such as "what was net recognized revenue last quarter?" and "how many active paying customers do we have?" Run them against the agent on every semantic layer change and on every agent or model upgrade.
Compare both the number and the metric the agent chose. An answer that is right by coincidence this month is still a wrong-definition bug. Our Quality Agent wraps rule-based test frameworks and anomaly detection, which gives you a place to schedule these checks and alert when a governed metric moves outside its expected range. The same idea works with a plain script in CI.
We also suggest asking a second question the golden set can answer: "What changed in the orders schema since last deploy?" Schema drift is a common reason a previously correct definition quietly breaks, and the Schema Agent surfaces column-level changes before the metric does.
What each failure looks like and what fixes it
| Symptom | Likely cause | Fix |
|---|---|---|
| Agent revenue is higher than finance's number | Summed gross amount, included refunds or test accounts | Governed metric with explicit filters (Step 2) |
| Agent answers differ between runs | Several equally plausible columns, model picks differently | Single named metric; remove ambiguous aliases (Steps 1-2) |
| Agent ignores the governed metric | Raw SQL tool is available and feels easier | Restrict tools to the semantic layer (Step 3) |
| Agent invents a definition when none exists | No rule allowing it to decline | Standing instruction to ask instead of guess (Step 4) |
| Number was right, then drifted | Schema or filter change upstream | Golden questions on every change (Step 5) |
Where this fits in a wider agent setup
Metric definitions are one slice of the context an agent needs. The same layer also carries lineage, ownership, PII classification and freshness. We covered how Data Workers keeps catalog context current in a separate post, and the principle holds here: if the context is stale, the agent is confidently wrong faster. If you are moving a data platform toward agent-driven operation, defining metrics once is a good first project because it is small, testable and immediately visible to finance.
Our Orchestration Agent can coordinate the pieces. When the Quality Agent sees a governed metric move, it can pull in the Schema Agent and the Incidents Agent to find the upstream cause. That is the point of running agents as a swarm rather than as separate point tools.
Frequently Asked Questions
Why does my AI agent pick the wrong revenue column? Because the warehouse contains several plausible columns and the agent chooses by name similarity and description, not by company policy. Without a governed definition it has no way to know which one finance reports.
Is a better prompt enough to stop an AI agent using the wrong metric definition? In our experience, no. A prompt that says "use net revenue" still leaves the agent to find and interpret the column, and it breaks when the schema changes. A semantic layer moves the definition into versioned code the agent calls, which is more reliable and reviewable.
Do I need dbt to do this? No. The dbt Semantic Layer is one well-documented option and the one we use in examples. Any semantic layer that exposes named, versioned metrics through an API or MCP server provides the same guarantee. What matters is one definition, one owner and one access path.
How do I know whether my agent is already giving wrong numbers? Run a golden question set against it. Pick questions finance has already answered, compare the agent's number and the metric it used, and treat any mismatch as a definition bug. If you have no confirmed answers to test against, that is itself the finding.
Can the agent still do exploratory analysis? Yes. Keep governed metrics for headline numbers and allow read-only exploration on marts for ad hoc work, but require the agent to label exploratory results as using no governed metric. Readers should always be able to tell the two apart.