Corrections Log Context Layer

A corrections log is a persistent store of every time a human told an AI agent it was wrong, fed back into retrieval so the same mistake never happens twice. It is the single highest-leverage addition to a data agent's context layer.

Most teams building data agents skip the corrections log because it sounds like infrastructure. It is — but it is the infrastructure that determines whether your agent learns from users or keeps repeating the same errors. This guide explains how to structure a corrections log, how agents consume it, and how to keep it from bloating. See also agent memory for data pipelines and AI for data infrastructure.

What Goes in the Log

Every correction has the same five fields. Store them as structured rows, not free text, so retrieval can filter precisely.

• •Question — the original natural-language query
• •Wrong answer — the SQL or table the agent picked
• •Correct answer — what the human said was right
• •Reason — one-line explanation of the mistake
• •Scope — which domain, tenant, or user the correction applies to
• •Timestamp — when it was logged, for decay

How the Agent Uses It

At query time the agent retrieves relevant corrections by semantic similarity to the incoming question. If a matching correction exists, the agent applies it before generating SQL. If multiple corrections conflict, the most recent and highest-scoped one wins. The corrections log becomes part of the context just like the schema and the glossary.

The trick is tight retrieval. A bloated log hurts more than an empty one. Filter by scope first, then by semantic similarity, then by recency. A tight filter means only one or two corrections enter the prompt per request, which keeps context small and accuracy high.

Capturing Corrections Cheaply

Corrections have to be cheap to capture or users will not bother. The simplest pattern is a thumbs-down button that opens a free-text field: tell us what the right answer was. A slightly smarter pattern asks a structured follow-up: which table should it have used, which join, which definition. Both work; the structured pattern produces better retrieval but requires more UI work.

A third pattern is implicit correction. When a user edits the SQL the agent produced and runs it, log the diff as a correction automatically. That requires no extra work from the user and produces a steady stream of high-quality corrections from power users.

Decay and Pruning

Corrections go stale. A table renamed last year is no longer relevant; a definition updated last quarter supersedes older entries. Corrections need a freshness score and a pruning policy. A good default is to weight corrections by recency, drop them below a threshold, and let humans override for corrections that should be permanent (legal definitions, policy rules).

Scoping Corrections

Not every correction applies to every user. A correction from the finance team about revenue definitions should not affect a product query about active users. Every correction needs a scope — global, domain, team, or user — and the agent should only apply corrections that match the scope of the current session.

Privacy and Security

Corrections can leak sensitive data if users type table names or values into them. The log must inherit the access control of the underlying warehouse: a correction from the finance team is only visible to finance agents. Logs should also redact free-text fields by default and surface them only to reviewers with permission.

Common Mistakes

The biggest mistake is treating corrections as a feedback signal for training the model. They are not — they are retrieval context. You feed them to the agent at query time, not into a fine-tune job. A fine-tuned agent still needs a fresh corrections log because corrections change faster than you can retrain.

Data Workers ships a corrections agent that captures, scopes, decays, and serves corrections to every downstream data agent. Teams that turn it on see accuracy improvements within a week as user feedback starts reaching the retrieval layer. To see it running, book a demo.

Measuring Impact

A corrections log is trivial to add to a demo and valuable only when it produces measurable accuracy improvements. The way to measure is simple: track accuracy on questions that matched a correction versus questions that did not. The delta is the corrections log value. Teams usually see 10 to 20 percentage points of improvement on matched questions, which is enormous.

The corrections log also pays off in user satisfaction. Users who see their corrections applied the next time they ask become advocates; users who see the agent repeat mistakes become detractors. The feedback loop from correction to applied fix should be as short as possible. Within one minute is ideal; within one day is acceptable; within one week means the log is not actually being applied.

Data Workers measures corrections impact continuously and exposes the metric in a dashboard. Teams can see the log paying off in real time and justify the engineering investment to leadership without hand-waving. Without measurement, corrections logs get deprioritized and decay.

Building the Capture UI

The capture UI is the single most important piece of corrections log infrastructure. If it is clunky, users do not correct and the log stays empty. If it is frictionless, corrections flow in and the log grows faster than it decays. Good capture UIs have two things: a single click to flag a wrong answer, and a single click to provide the right answer.

The best UIs go further and let users edit the generated SQL directly. When the user edits and runs, the diff becomes the correction automatically. No separate step, no extra friction. Power users produce dozens of corrections a week this way, and every one feeds back into retrieval.

The UI also needs to show users that their corrections are being applied. When the same user asks a similar question later, the agent should surface the correction: applied your correction from last week about fct_revenue vs stg_orders. That positive feedback loop encourages more corrections and builds trust in the system.

Corrections logs are the context-layer equivalent of unit tests: cheap to add, impossible to live without once you have them. Capture corrections, scope them, decay them, and feed them to retrieval, and your agents learn from every user.