How Do You Trace a Failed Tool Call or Retry That an AI Agent's Successful Response Hides?

To trace a failed tool call or retry that a successful agent response hides, record every tool invocation and every retry attempt as its own span with its own status, and stop treating the final answer as the evidence. The MCP specification, revision 2025-06-18, lets a tool result carry an isError flag separate from the protocol response, so a failed call can sit inside a perfectly healthy conversation. This guide covers what to capture, how to correlate it with pipeline runs, and where a human still decides.

Key Takeaways

• •A successful agent response proves the model produced text, not that every tool call it made succeeded.
• •Log each tool call and each retry attempt as a separate span with its own status, arguments hash, duration and error.
• •Carry one trace ID from the agent through the MCP server into the pipeline run, so the agent's claim can be checked against what the warehouse and orchestrator actually did.
• •Alert on divergence between the final answer and the span tree, such as an answer that cites data from a call that errored.
• •Retries are a fix for transient faults only; a retry that changes the result must be visible to the reader of the answer.

Why a successful response hides a failed tool call

An agent response is a summary written by a model. Between the question and that summary sit several tool calls: a schema lookup, a query, a freshness check. If one of them fails, the model sees an error string in its context, may retry, may fall back to a cached value, and then writes a fluent answer. Nothing about the final text has to reveal any of that. The problem was put well in a recent post on X asking how you see what actually ran when a success hides a failed call and a retry.

Two layers each report success for different reasons. The transport layer reports success because the request was delivered and a result came back. The model layer reports success because it generated an answer. Neither layer is lying. They are answering a narrower question than the one the reader of the answer cares about, which is whether the data behind the sentence was actually read.

In data work the cost is concrete. A freshness check that timed out, followed by a retry that read a stale replica, produces an answer about yesterday's table stated in today's confident tone. The engineer reading it has no reason to doubt it. The only defense is a record of what ran, kept outside the model's own narration of events.

What to record for every tool call

Start with the unit of record: one span per tool call. The OpenTelemetry GenAI semantic conventions define span attributes for model and agent operations, and they are a sensible naming base so your traces open in ordinary tracing backends. Do not invent a private schema if a shared one covers the field. Add data-specific attributes on top: the target warehouse, the table or asset, and the pipeline run the call relates to.

Capture the outcome as data, not as prose. A status field with a small closed set of values, an error class, an error message truncated to a safe length, and the duration. Hash the arguments rather than storing them raw, because tool arguments in data work often contain filter values that are themselves sensitive. A hash still lets you tell whether two attempts were the same call or a changed one.

Keep the list short enough that engineers will actually fill it in. The minimum we would record on every call:

• •Trace ID, span ID and parent span ID, so the call hangs off the user's question.
• •Tool name, MCP server name and the target asset or table.
• •Attempt number and a reason code if it is a retry.
• •Status, error class and duration in milliseconds.
• •A hash of the arguments and a hash of the returned payload, so identical retries and identical results are detectable.

Record each retry attempt as its own span

The most common blind spot is collapsing retries into one logged event. If the log line says the call succeeded in 4.2 seconds, you cannot tell whether that was one attempt or three. Record every attempt as a sibling span under the same parent, numbered, with the failure reason of the previous attempt on the next one. Then a successful call that needed three tries looks different from a clean one, even though both end with status success.

Know which layer is retrying, because there are usually several. The orchestrator may retry a task, the MCP client may retry a request, and the model may decide to call the tool again after reading an error. Airflow's configuration reference lists default_task_retries with a default of 0, which means any retry in an Airflow-run pipeline is something your team configured. Audit where each layer's retry policy is set, since stacked retries multiply the attempts and the load.

Separate transient retries from semantic ones. Retrying after a timeout is the same request again. A model that rewrites the query after an error and calls again is making a new call, and it may be changing the meaning of the question. Tag the second kind differently, because a changed query that returns data is the case where the answer can drift furthest from what the user asked.

Carry one trace ID from the agent to the pipeline run

A trace that ends at the MCP server answers only half the question. The other half is what the warehouse and the orchestrator did in response. Pass the trace ID as a query tag on warehouse statements and as a run-level parameter or label on orchestrator runs. Then a span that says the tool called the warehouse can be joined to the query history row and the task instance, using the same identifier, with no guesswork about timestamps.

This join is what lets you test the agent's claim. If the answer says the orders table was fresh as of 06:00, the trace should lead to a query that read the freshness metadata and a result that said so. If the trace leads to a query that errored, or to no query at all, the claim has no backing. That check is mechanical once the identifier exists, and nearly impossible without it.

Keep the identifier propagation in the framework rather than in each tool. When every tool author has to remember to forward the ID, some will forget, and the missing spans will be exactly the calls nobody wrote tracing for. Put it in the client wrapper that every agent goes through, so a call without a parent trace cannot be made.

A worked example: a green answer over a stale read

Here is an illustrative case, not a customer incident. An engineer asks, "Show freshness scores for all gold tables." The agent calls a metadata tool for each table. The call for one table times out, the client retries, and the retry is served from a cached result written two days earlier. The agent reports all tables fresh. The response is fluent and every table has a score.

With only response logging, this looks like a success. With per-attempt spans, the trace shows attempt one with status timeout, attempt two with status success and a duration near zero, and a payload hash that matches an earlier trace. A near-zero duration plus an old payload hash is the signature of a cache hit standing in for a live read. A simple rule can flag it automatically.

The fix has two parts. The tool should return the age of its data, so a stale value is labeled stale rather than passed off as current. The agent's instructions should require it to state when a score came from a retry or a cache. Tracing found the problem; the contract change prevents the next one from hiding in the same place.

Alert on divergence, not only on errors

Error-rate alerts on tool calls are necessary and not enough, because the dangerous case is an error followed by a recovery. Alert instead on divergence: spans in a trace with a non-success status where the final answer does not mention a limitation, retried calls whose results differ from the first successful result for the same arguments hash, and answers that cite assets with no successful read span in their tree.

Treat the MCP isError flag as a first-class signal. The MCP tools specification distinguishes protocol errors from tool execution errors, which are returned inside a normal result. A client that only watches for protocol failures will see a tool that failed gracefully as a success. Map the execution-error flag onto your span status explicitly.

Start with a handful of rules and review what they catch each week. Too many divergence alerts will teach the team to ignore them. Rules that fire rarely and are right when they do will get read, and each false positive is a prompt to tighten the rule or fix the tool that produced ambiguous output.

Where a human still signs off

Tracing tells you what happened. It does not decide whether an answer built on a retried call is acceptable. That depends on the use. A freshness answer feeding a dashboard annotation tolerates a stale read differently from one feeding a financial close. Set the policy per domain: which tools may retry, how many times, and whether a retried result may be presented without a caveat.

For anything that writes, be stricter. A retry on a read is a nuisance when it goes wrong. A retry on a write can apply a change twice. Require idempotency keys on write tools and put an approval step before the first attempt rather than after a failure. Reviewing the span tree should be part of that approval, since it shows what the agent already tried.

Keep a person in the loop for the review of the rules themselves. Divergence rules encode a judgment about what counts as a hidden failure, and that judgment changes as pipelines and agents change. A short weekly look at flagged traces, with a decision to keep, tighten or retire each rule, keeps the system honest.

How Data Workers handles this

Every Data Workers agent is an MCP server, so every capability is a tool call that can be traced the same way. The Orchestration Agent coordinates work across the other agents and keeps one trace across the handoffs, so a question that touches several agents produces a single span tree rather than several unrelated logs. Calls and retries are recorded as separate attempts with their own status.

When a pipeline run is involved, the Pipeline Agent and the Incidents Agent work from the same identifier, so the agent's claim about a run can be checked against what the orchestrator logged. Where a call failed and a retry covered for it, that shows up in the incident context instead of disappearing behind the final answer.

The open-source edition is read-only by default and runs locally, which makes it a low-risk place to try this: point it at a warehouse, ask a question you already know the answer to, and read the span tree. Writes, approval workflows and audit trails arrive in the paid editions, with data staying in your own infrastructure.

Frequently Asked Questions

Why does my AI agent report success when a tool call failed?

The agent reports what the model wrote, and the model can recover from an error by retrying, switching tools or relying on cached context before it writes the answer. The transport also reports success when a tool returns an error result inside a normal response. Success at those layers does not mean every call succeeded. Record per-call status and compare it with the answer.

Should I log tool arguments and results in full?

Usually not. In data work, arguments and results can contain personal or sensitive values, and full payloads make traces expensive and risky to share. Store hashes, sizes, the target asset and a truncated, redacted error message. Hashes still let you detect identical retries and identical results, which is what most debugging needs. Keep full payloads only in a restricted store with retention limits.

How do I tell a transient retry from the agent changing its query?

Compare the arguments hash across attempts under the same parent span. The same hash means the same request was sent again, typically after a timeout. A different hash means the agent rewrote the call, which can change what the answer means. Tag the two cases with different reason codes and review the second kind more closely.