How to Stop AI Agents From Shipping Breaking Schema Changes That Still Pass dbt build
A five-layer guard for the failure mode where the build is green and the contract is broken
To stop an AI agent from shipping a breaking change that still passes dbt build, enforce dbt model contracts on every model other teams consume, version those models, and fail CI on any state-based diff that changes a contracted interface. A passing build only proves that the SQL compiles and your tests pass. It does not prove that the output still means what downstream consumers think it means. This post lays out five layers that close that gap.
Key Takeaways
- •A green dbt build proves compilation and test success, not backward compatibility with downstream consumers.
- •dbt model contracts catch removed columns, renamed columns and type changes at build time, but only on models where contract enforcement is switched on.
- •Contracts cannot catch semantic changes, such as a new filter or a redefined metric, so those need a diff review and a consumer check.
- •Agents make many small changes quickly, so the guard has to be automated in CI rather than depending on a human reading each diff.
- •Breaking changes should ship as a new model version with a deprecation window, never as an in-place edit.
Why a passing dbt build can hide a breaking change
The failure mode is simple. An agent is asked to clean up a model. It renames a column, narrows a join, or changes a cast. The model compiles, the generic tests on the columns that still exist pass, and dbt build exits zero. Meanwhile a dashboard, a reverse ETL sync and two other teams' models now read nulls, drop rows, or fail on the next run. Nothing in the build knew those consumers existed.
An engineer who writes the change by hand usually carries that context in their head. They remember that finance reads the column, or that the sync runs at 6am. An agent has only what it was given in the prompt and the repo. The dev.to write-up on designing schema boundaries for AI agents makes the same point: human review fails once agents make repeated changes, and a passing build can hide a breaking change.
There are two distinct problems hiding in that sentence. The first is structural drift: columns, types and constraints that change shape. The second is semantic drift: the shape stays identical but the values mean something different. Tools for the first problem are mature. The second needs more work, and we cover both below, because an agent can cause either one in a single commit.
The five layers that catch it
We treat this as a defence in depth problem. No single check is enough, because each one covers a different kind of break. The table below maps each layer to what it catches and what it still misses, so you can see which gaps remain if you only adopt one or two.
| Layer | What it catches | What it misses |
|---|---|---|
| 1. Model contracts | Removed or renamed columns, type changes, dropped constraints | Changed logic or filters with the same shape |
| 2. Model versions | In-place edits to a public interface without a migration window | Models nobody marked public |
| 3. State-based CI diff | Any modified node, including contract-affecting edits | Consumers outside the dbt project |
| 4. Consumer blast radius | Dashboards, syncs and external readers of a changed column | Changes with no column-level lineage |
| 5. Approval and audit | Unreviewed agent writes to protected models | Nothing, if the earlier layers are skipped |
Read the table top to bottom as an order of adoption. Layers 1 to 3 live entirely in your dbt project and CI and cost very little. Layers 4 and 5 need lineage and an approval path, which is where most teams have gaps today.
Layer 1: enforce contracts on every public model
A dbt model contract declares the columns and data types a model promises to return. When you set contract: {enforced: true} on a model, dbt checks the built output against that declaration and fails the build if they disagree. That turns a silent rename or a type change into a hard failure at the exact moment the agent makes it, which is the cheapest place to catch it.
Contracts also let you declare constraints such as not null, and the dbt documentation on model contracts describes which constraints your warehouse actually enforces and which dbt only records. Check that page for your adapter before you rely on a constraint as a safety net. A constraint the warehouse ignores is documentation, not protection.
The rule for agents is blunt: any model with an external consumer gets a contract, and the agent is not allowed to edit the contract block and the model SQL in the same change without a human approval. If the agent can loosen the contract to make the build pass, the contract protects nothing. Keep the contract files in a path your CI treats as protected.
Layer 2: version public models instead of editing them
Contracts tell you that a break happened. Versions tell you how to make a break safely. dbt supports model versions, where a breaking change ships as a new version such as dim_customers_v2 while the previous version stays available. Consumers move on their own schedule, and you can set a deprecation date on the old version.
For agents, the useful rule is that a diff touching a contracted public model in a breaking way must create a new version, not mutate the existing one. This is easy to express as a CI check and easy for an agent to follow, because the instruction is mechanical. It also gives the human reviewer a natural question to answer: who is on v1, and when do they move?
Be honest about the cost. Versioning means two models to maintain during the window, and it needs someone to own the migration. For models with one internal consumer, a coordinated change in a single pull request is often simpler. Reserve versions for models that cross team boundaries.
Layer 3: fail CI on state-based diffs
Contracts fire on build, but you want the signal earlier, on the pull request. dbt can compare the current project against a saved manifest from your main branch and select only what changed, using the state:modified selector. In CI, build the modified nodes and their children, and add a step that lists every modified model carrying a contract or a public access setting.
That list becomes the review gate. If the agent's change touches no contracted model, it can merge on green. If it touches one, CI posts the list and blocks until a named owner approves. The open-source dbt-core repository is the reference for how state comparison behaves, and it is worth reading the issues on false positives before you rely on it, since comments and whitespace can affect what counts as modified.
Semantic drift is where this layer helps most. A diff that adds a where clause changes no column names, so Layer 1 stays quiet, but the state diff still flags the model as modified. Pair that flag with a data diff on a sample of keys, and the reviewer sees row-count and value changes before merge instead of after.
Layer 4: check the blast radius before the merge
State comparison only knows about nodes inside the dbt project. Many real consumers are outside it: BI dashboards, reverse ETL jobs, notebooks, and other teams' pipelines. Before an agent merges a change to a contracted model, something needs to answer the question we hear most often: what is the blast radius if I drop or change this column?
That answer comes from column-level lineage plus query history. Query history is also your safety net on the warehouse side. For example, Snowflake's Time Travel documentation states that the default retention is 1 day for standard objects and can be extended up to 90 days on Enterprise edition, which bounds how long you have to recover a table an agent overwrote. Check your own retention setting, because the default is shorter than most incident timelines.
Our Schema Agent does this projection for column changes: it detects drift, lists downstream models and reports the consumers it can see, then proposes a migration. We would still tell you to verify its output against your own lineage the first few times, since any lineage tool has blind spots for dynamic SQL.
Layer 5: put agents behind approval tiers and an audit trail
The last layer is about permissions, not parsing. Decide in advance which actions an agent may take alone, which need approval, and which it may never take. A workable split is: read-only analysis and drafting always allowed, edits to non-contracted models allowed with CI, edits to contracted models require an owner's approval, and edits to contract files themselves are never automatic.
Every agent action should land in an audit log that records the prompt, the diff, the checks that ran and the approver. The Model Context Protocol specification defines how clients and servers exchange tool calls, but it does not define approval or audit behaviour, so you have to build that policy layer yourself or use a platform that provides it.
This is also where the human stays useful. A reviewer should not read every diff. They should read the short list of diffs that crossed a protected boundary, with the blast radius attached. That keeps review sustainable even when an agent opens a dozen pull requests in a day.
A rollout order that works
Do not try to install all five layers at once. Start by listing the models with external consumers and adding contracts to those alone. This is usually a small fraction of the project, and it gives you most of the structural protection immediately. Then add the CI step that lists modified contracted models.
Next, add versioning for the two or three models that cause the most coordination pain, and tighten agent permissions so contract files are protected paths. Only then invest in column-level lineage for the blast radius check, since it is the most work and depends on what your warehouse and BI tools expose.
Finally, measure it. Track how many agent changes touched a contracted model, how many were blocked, and how many blocks were correct. If CI blocks nothing for a month, either your agents are well behaved or your checks are not wired up. Test that by deliberately renaming a column on a branch and confirming the pipeline fails.
Where the swarm approach helps
Each layer above can be built from separate tools, and many teams do. The cost is that the engineer becomes the integration layer, stitching together a CI result, a lineage view and an approval ticket. In our design, the Schema Agent maps the blast radius while the Pipeline Agent proposes the versioned fix, and both share the same context about what is protected.
That coordination is the argument for a swarm, but it is not magic. If you have strong dbt discipline, contracts plus a state diff in CI will carry you a long way without any agent platform. Where we add value is the cross-system part: lineage outside dbt, approval routing, and an audit trail that follows the change from prompt to merge.
Frequently Asked Questions
Does dbt build fail when an agent removes a column?
Only if the model has an enforced contract or a downstream model or test references the column. Without a contract, dbt builds the model with whatever columns the SQL now returns. Downstream models that select the missing column will fail, but BI tools and external consumers will not be checked at all. Enforcing contracts on public models is the direct fix.
Can model contracts catch a change in business logic?
No. A contract checks names, types and constraints, so a new filter or a redefined calculation with the same output shape passes. Catch these with a state-based diff that flags the modified model, a sampled data diff, and a human owner who approves changes to metrics other teams depend on.
Should an AI agent be allowed to edit contract files?
We recommend not. If an agent can edit both the model and its contract in one change, it can make a breaking change look compliant. Treat contract files as protected paths that require a named owner's approval, and have the agent propose a new model version instead when a break is intended.
Do we need model versions for every model?
No. Versions add maintenance overhead, so use them for models that cross team boundaries or feed external systems. For models with a single internal consumer, a coordinated change in one pull request, backed by a contract and CI, is usually enough.