Why data agents need column-level lineage to act safely

Data agents need column-level lineage because the unit of damage in a warehouse is a column, not a table: dropping, renaming or retyping one column can break a dozen downstream models while the other forty columns in the same table are untouched. The OpenLineage specification defines column-level lineage as a facet precisely because table-level edges cannot answer 'which output fields are derived from which input fields.' An agent that edits SQL, migrates schemas or reruns pipelines without that answer is guessing, and it guesses faster than a human does.

Key Takeaways

• •Table-level lineage says two tables are connected; column-level lineage says which fields flow through and how they are transformed. Only the second answers 'what breaks if I change this?'
• •An agent without column-level lineage has two bad options: refuse every change, or approve changes it cannot verify. Neither is useful.
• •PII governance is a column-level problem. A masked column in one table is worthless if an unmasked copy flows into a mart.
• •Column-level lineage is an input to the agent's decision, not a replacement for approval. Writes should still sit behind a human-controlled autonomy setting.
• •Lineage is only as good as its freshness. Stale lineage is worse than none because it produces confident wrong answers.

Why data agents need column-level lineage to act safely

A human engineer who is asked to drop a column does a lot of unwritten checking. They search the repo, ask in Slack who uses it, remember that finance has a dashboard built on it, and hesitate. An agent running in Claude Code or Cursor has none of that tribal memory. It has the context we hand it, and the quality of the decision is bounded by the quality of that context. Lineage is the most important piece of it.

Consider the question we hear most often from teams adopting coding agents: 'What's the blast radius if I drop this column?' With table-level lineage, the honest answer is 'every model that reads this table,' which in a mature warehouse is a long list, most of which does not touch the column at all. The agent either flags all of them (noise, so people stop reading) or none of them (silence, so something breaks). With column-level lineage, the answer is the specific set of downstream columns, models, dashboards and exports that actually derive from the field.

What goes wrong without it

We see the same four failure modes whenever an agent touches a warehouse without field-level dependency data.

• •Over-blocking: every change touching a popular table looks risky, so the agent asks for approval on everything and the team turns the guardrail off.
• •Silent breakage: a rename passes tests on the changed model and fails three hops downstream, where a join key no longer matches.
• •Semantic drift: a column keeps its name and type but changes meaning (a timezone, a currency, a filter), and nothing in a table-level graph records that.
• •Policy leaks: a PII column is tagged at the source, but a derived column built from it carries the same data into an ungoverned schema.

The silent-breakage case is the one that costs the most. A question like 'Is the customer_id join safe after the migration?' is only answerable if the agent can see every place customer_id is used as a join key, and whether each consumer expects the old or new type.

Column grain is where governance actually lives

Access policy, masking, retention and classification are all attached to columns. When someone asks 'Who has access to PII columns in analytics?', the answer depends on propagation: a column derived from an email address through a hash, a concatenation or a join is still sensitive in most policies, and a table-level graph cannot tell you that. Column-level lineage lets a governance check follow the data instead of the table name.

This is also why we treat lineage as shared context rather than a feature of one tool. Our Catalog Agent, part of the Data Workers platform, federates metadata from the catalogs a team already runs, and the lineage it reads is available to the quality, governance and schema agents in the same session. We covered how that shared context keeps the catalog current in a separate post. The point here is narrower: an agent that proposes a change should read the same dependency graph that the governance check reads.

What an agent should ask before it writes

We think a safe data agent runs a fixed pre-flight on any change that touches a column. None of this is exotic; it is what a careful engineer does, written down so it runs every time.

• •Resolve the target: which exact column, in which table, in which environment.
• •Walk downstream at column grain: which derived columns, models, dashboards, reverse-ETL syncs and notebooks depend on it, and through which transformations.
• •Check classification: does the column, or anything derived from it, carry PII or a retention policy.
• •Check freshness of the lineage itself: when was the graph last refreshed, and does it cover the consumer types that matter (BI tools, exports, ad hoc notebooks).
• •Decide by autonomy level: auto-apply, propose a diff for review, or stop and escalate.

The last step matters as much as the first four. Lineage tells the agent what is affected; it does not decide whether the team is comfortable with an agent acting alone in that domain. We expose that as a per-domain autonomy setting in the Autonomous Data Conductor, so a team can let the agent auto-apply low-risk changes in a sandbox schema while requiring review for anything that reaches a governed gold table.

How this works over MCP

The Model Context Protocol gives coding agents a standard way to call tools, which is why lineage can be a tool call rather than something pasted into a prompt. In Claude Code, an engineer can ask a question in plain language and the agent calls the lineage tool, gets structured dependencies back, and reasons over them. Every Data Workers agent is an MCP server, so the same call works from Claude Code, Cursor, Windsurf, VS Code or any other MCP-compatible client.

In practice the useful prompts are the ones engineers already ask: 'What changed in the orders schema since last deploy?', 'What's the blast radius if I drop this column?', 'Is the customer_id join safe after the migration?' Our Schema Agent maps the affected columns; when a change has already broken something, the Incidents Agent uses the same graph to trace root cause upstream instead of searching logs blindly.

Where Data Workers fits, and where it does not

We should be plain about the limits. Data Workers does not replace the system that emits lineage: column-level lineage still has to be produced by your transformation layer, your orchestrator or a parser of your SQL, and where it cannot be inferred (dynamic SQL, opaque UDFs, spreadsheets) the graph has gaps. Our agents read and combine what exists, flag where coverage is missing, and refuse to present a partial graph as complete.

What we add is the coordination. The open-source edition (dw-claw) is Apache 2.0, runs locally and is read-only by default, so a team can try the lineage-aware questions above without granting any write access. Pro adds writes, approval workflows and audit trails for teams ready to let agents apply changes. That order is deliberate: let the agent explain before it acts.

One caveat is our own opinion rather than a measurement: we think most agent mishaps in data engineering will come from missing context, not from weak models. A strong model with a wrong dependency graph will make a confident, well-written, incorrect change.

Frequently Asked Questions

What is column-level lineage? It is a record of which input columns each output column is derived from, and through what transformation. Table-level lineage records only that one table feeds another.

Why is table-level lineage not enough for an AI agent? Because it overstates impact for most changes and understates it for the dangerous ones. An agent cannot tell a harmless edit from a breaking one when every column in a table looks equally connected.

Can an agent generate column-level lineage on its own? It can help by parsing SQL and reading orchestrator metadata, but it cannot infer dependencies hidden in dynamic SQL or external tools. Good agents report those gaps instead of filling them with guesses.

Does column-level lineage replace human approval? No. Lineage informs the decision. Whether the agent may apply a change without review is a policy choice, and we recommend setting it per domain, starting read-only.

How do I try this with Claude Code or Cursor? Install the open-source dw-claw edition, connect the MCP servers to your client, and ask a question such as 'What's the blast radius if I drop this column?' against a non-production environment first.