Why Does More Context Make a Data Agent Less Accurate?

More context makes a data agent less accurate because every extra token competes for the model's attention, and irrelevant schema, stale memory and conflicting instructions bury the few facts the question needs. The research is consistent: Liu et al. found that models use information in long contexts unevenly, and performance drops when the relevant fact sits in the middle of the input. A practitioner version of the same finding surfaced recently, when an engineer reported that "we gave our finance agent more context and it got worse" in a talk at AI Engineer NY.

Key Takeaways

• •Context is a budget, not a bucket: each token you add must earn its place against the question being asked.
• •Accuracy falls when relevant facts are diluted, buried mid-prompt, or contradicted by stale memory, even when the window is nowhere near full.
• •A useful context layer selects per question: it scopes to the relevant tables, ranks by relevance, and caps what enters the prompt.
• •Memory needs expiry and ownership, otherwise yesterday's schema becomes today's wrong answer.
• •Treat every context change like a code change: run it against an eval set before it ships.

Why more context can make a data agent worse

The intuition that a bigger window means a better answer is wrong in practice. Context windows are large: the Anthropic documentation on context windows describes a standard window of 200,000 tokens for current Claude models. That is room for thousands of table definitions. But capacity is not the same as attention. A model can hold the text and still fail to weigh the one column description that decides the answer.

Chroma's Context Rot research tested how models behave as input length grows and reported that performance degrades with longer inputs even on simple tasks. For data work this matters more than for chat, because warehouse metadata is repetitive. Fifty tables with a column called status or amount look nearly identical to a model, and the agent picks the wrong one with full confidence.

We see three failure shapes. The first is dilution: the right table is present but surrounded by forty near-duplicates. The second is conflict: two documents define "active customer" differently and the agent blends them. The third is staleness: a remembered fact about a schema that changed last week. All three get worse as you add context without curating it.

Layer 1: scope context to the question

The first layer decides what the agent is allowed to see for this specific question. A question about revenue by region needs the orders and region tables, their joins, and the metric definition. It does not need the marketing event tables or the HR schema. Scoping starts from the question, resolves it to a small set of candidate assets, and loads only those.

Anthropic's engineering guidance on effective context engineering for agents frames the goal as finding the smallest set of high-signal tokens that makes the desired outcome likely. For a data agent, that means retrieval by domain, ownership and lineage rather than dumping the full information schema into the system prompt.

A practical test: for ten real questions, list by hand which tables a good analyst would open. If the agent's prompt contains ten times that number, scoping is your first fix. Our Catalog Agent, available through Data Workers, does this resolution step so the model receives candidates, not the whole catalog.

Layer 2: rank and cap what enters the window

Scoping narrows the field; ranking orders it. Even among relevant assets, some carry more signal: certified tables beat scratch tables, recently queried tables beat dormant ones, and tables with documented columns beat undocumented ones. Rank on those signals and set a hard cap on how many assets and how many tokens per asset can enter the prompt.

Position matters too. The lost-in-the-middle findings show that models attend best to the start and end of the input. Put the question, the top-ranked asset and the governing metric definition at the edges, and keep lower-ranked material short. A cap also gives you a number to tune, which makes the next layers testable.

Resist the urge to raise the cap when an answer is wrong. Wrong answers usually mean the ranking failed to surface the right asset, not that the window was too small. Raising the cap hides the ranking bug and adds dilution.

Layer 3: separate facts from instructions

Agents get worse when facts, rules and examples are mixed into one block. Schema facts (column types, row counts, lineage) change with the warehouse. Business rules (how we define churn) change with the company. Instructions (always filter test accounts) change with the task. Store them separately, version them separately, and load each only when relevant.

This also removes conflicts. If a metric definition lives in exactly one governed place, the agent cannot read two competing versions. When the definition changes, one edit fixes every future answer. We covered the underlying idea in our piece on what an AI agent needs to know about your warehouse, where the context layer holds schema, lineage, ownership, policy and cost as distinct inputs.

Instructions deserve the strictest budget. Long rule lists read like policy but behave like noise; each additional rule lowers the odds that any single one is followed. Keep the instruction block short and move enforcement into tools, such as a query validator, where a violation fails loudly instead of being quietly ignored.

Layer 4: design memory with expiry and ownership

Memory is where context grows without anyone deciding to grow it. An agent that saves every correction, every preference and every discovered join accumulates thousands of notes. Some are right, some were right once, and some were never right. Without curation, retrieval from memory returns a pile of plausible but conflicting history.

Give every memory entry an owner, a source and an expiry. Facts derived from the schema should expire when the schema changes, which you can detect from the catalog. Human corrections should carry the name of the person who made them and be reviewed on a schedule. Entries that are never retrieved should be dropped, since unused memory is pure cost.

Prefer memory that stores pointers over memory that stores copies. A note that says "the revenue metric is defined in the governed metrics store" cannot go stale. A copied definition can. This keeps the agent's working context small and the source of truth singular.

Layer 5: evaluate context changes like code

The only reliable way to know whether context helps is to measure it. Build an eval set from real questions your team asks, such as "why did row counts drop 40% overnight?" or "show freshness scores for all gold tables." For each, record the correct answer or the correct set of tables. Every change to scoping, ranking, memory or instructions is then run against that set before it ships.

Track accuracy, but also track context size and which assets were retrieved. When accuracy drops after you add something, the retrieval trace tells you whether the new context displaced the right asset or contradicted it. When accuracy holds while tokens fall, you have found free savings.

Run ablations. Remove one context source at a time and re-run the set. Teams are often surprised to find a document everyone assumed was essential has no effect, or a negative one. That is the practical lesson behind the reports of agents that got worse after being given more: nobody measured the addition.

How Data Workers applies the five layers

Data Workers agents are MCP servers, so context arrives through tools the model calls on demand instead of a giant preloaded prompt. The Model Context Protocol specification defines how tools and resources are exposed, which lets an agent pull a table's lineage or owner only when the question needs it. That is Layer 1 and Layer 2 by design.

Because the agents share one context across catalog, quality, schema and governance, a metric definition or a policy lives in one place and every agent reads the same version. That covers Layer 3. Staleness detection comes from the schema and catalog agents noticing changes, which feeds the expiry rules in Layer 4. You can run your own eval set against the same tools from Claude Code or Cursor.

Frequently Asked Questions

Does a bigger context window fix accuracy problems?

No. A larger window raises how much you can include, not how well the model uses it. Research on long-context use shows accuracy depends on where information sits and how much surrounds it. Curate first, and only then consider a larger window for tasks that genuinely need long inputs.

Should we use retrieval or put the whole schema in the prompt?

Use retrieval for anything beyond a small schema. Pass the agent a ranked, capped set of relevant assets and let it request more through tools. Loading everything is simple to build and expensive to debug, because you cannot tell which of the thousand tables caused the wrong join.

How do we know which context to remove?

Run ablations against an eval set of real questions. Remove one source, such as a document, a memory store or an instruction block, and compare accuracy and token count. If accuracy does not fall, the source was not helping. Keep the evals in version control so the result is repeatable.