How to Review Data Changes in dbt Pull Requests

A dbt pull request can pass every test, compile cleanly, and still ship wrong numbers. Code review tells you whether the SQL is readable; it does not tell you what the SQL does to the data. This is the practitioner's playbook for reviewing a dbt PR's data impact before merge — cheaply and systematically.

The core idea is a review ladder: start with checks that cost nothing (lineage and schema diffs), escalate to row counts, then column profiles, and only pay for a row-level value diff when the cheaper rungs say the change is risky. Most PRs never need the expensive rung. The ones that do are exactly the ones that would have burned you.

Why Green CI Does Not Mean Correct Data

Standard dbt CI answers one question: does the project build and do the declared tests pass? That is necessary and nowhere near sufficient. Schema tests like not_null and unique assert structural invariants, not business correctness. A model can satisfy every test you wrote and still produce a materially different answer than it did yesterday.

Three failure modes show up constantly in real dbt PRs. First, type casts that round differently: changing NUMERIC(38, 4) to FLOAT, or moving a division before an aggregation instead of after it, shifts totals by fractions of a percent — small enough to pass tests, large enough to matter when finance reconciles the quarter. Second, joins that silently drop or duplicate rows: switching a LEFT JOIN to an INNER JOIN to fix a null, and quietly losing every order whose customer record has not landed yet; or joining on a key that turned out not to be unique, fanning out revenue rows. Third, the refactor-only PR that is not: a description says no behavior change, but extracting a CTE moved a WHERE filter inside an aggregation boundary and the revenue mean shifted by two percent. None of these break compilation. None of them necessarily break a test. All of them break trust in the numbers.

The fix is not more unit tests — it is comparing the data the PR produces against the data production currently produces, with escalating precision.

The Review Ladder

Each rung answers a sharper question at a higher cost. Run them in order and stop as soon as you have enough evidence for a verdict.

Rung 1: Lineage and Schema Diff (Free)

Before touching the warehouse, diff the two dbt manifests. Comparing manifest.json from the PR branch against production tells you which models were modified, which columns were added, removed, or retyped, and — critically — the downstream blast radius. A change to a staging model with forty downstream marts deserves a different level of scrutiny than a leaf-node report model. dbt ls --select state:modified+ --state path/to/prod-artifacts gives you the modified set plus everything downstream in one command.

Schema diffs catch an entire class of bugs by themselves: a column that silently changed from TIMESTAMP to DATE, a renamed column that a downstream BI tool references by name, a dropped column nobody mentioned in the PR description. If the schema diff is empty and the model is a leaf with no downstream consumers, you can often stop here.

Rung 2: Row Counts (Cheap)

Build the PR branch into its own schema (more on that below), then count both sides: SELECT COUNT(*) FROM prod.fct_orders versus SELECT COUNT(*) FROM pr_1234.fct_orders. Compute the percent delta. A refactor-only PR should produce a delta of exactly zero; any nonzero delta on a supposedly behavior-neutral change is an automatic escalation to rung 3. For PRs that intentionally change the population — a new filter, a backfill — the question becomes whether the delta matches the author's stated expectation. A PR that claims to remove test accounts and drops 0.3% of rows is plausible; one that drops 14% needs an explanation before merge.

One subtlety: when the base table is empty, the percent delta is undefined, not zero. Report it as such — collapsing it to zero hides exactly the new-model and rebuilt-model cases where you have no baseline and should be more cautious, not less.

Rung 3: Column Profiles (Moderate)

Row counts can match while the contents drift. A column profile compares per-column statistics across environments in a single aggregate pass: SELECT COUNT(*) AS row_count, COUNT(amount) AS non_null, COUNT(DISTINCT amount) AS distinct_vals, MIN(amount), MAX(amount), AVG(amount) FROM ... run against both schemas, then diffed. For categorical columns, compare null rate and distinct count; for numerics, add min, max, and mean; for timestamps, min and max tell you whether the date range shifted.

This is the rung that catches the type-cast rounding bug and the shifted revenue mean. If AVG(revenue) moved on a PR whose description says refactor only, the PR description is wrong, and the review conversation changes from style to correctness. Profiles are also where null-rate regressions surface: a join change that turned 0.1% nulls into 9% nulls is invisible to a row count and obvious in a profile.

Rung 4: Row-Level Value Diff (Expensive, Decisive)

When the cheaper rungs flag something, find out exactly which rows changed. The workhorse pattern is a FULL OUTER JOIN on the primary key with a row classification: SELECT COALESCE(p.order_id, b.order_id) AS order_id, CASE WHEN p.order_id IS NULL THEN 'added' WHEN b.order_id IS NULL THEN 'removed' WHEN p.amount IS DISTINCT FROM b.amount THEN 'changed' ELSE 'identical' END AS diff_status FROM prod.fct_orders p FULL OUTER JOIN pr_1234.fct_orders b ON p.order_id = b.order_id. Aggregate diff_status for the summary, and sample a few dozen changed rows for the reviewer to inspect by eye. Use null-safe comparison (IS DISTINCT FROM, or BigQuery's IS NOT DISTINCT FROM negated) so null-to-null does not register as a change.

Cap the output. You want the classification counts plus a bounded sample of changed rows — not a billion-row diff dumped into a PR comment. On large models, run the diff on a deterministic sample of keys first and only widen if the sample looks wrong.

Wiring It Into CI

The whole ladder automates cleanly. On every PR: build the modified models (plus first-order children) into a dedicated schema named after the PR, using dbt build --select state:modified+ --defer --state prod-artifacts so unmodified parents resolve to production tables instead of being rebuilt. Then run the ladder against the prod-built base: manifest diff, row counts, profiles on touched columns, and a PK value diff only for models the earlier rungs flagged. Post one summary comment on the PR — counts, deltas, profile shifts, a sample of changed rows — and emit a machine-readable risk verdict (pass, warn, block) that the merge gate can act on. A warn verdict means a human must acknowledge the data change; a block means the numbers contradict the PR description.

Two operational details matter. Clean up PR schemas on merge or close, or your warehouse fills with orphans. And pin the production artifacts you compare against to the commit actually deployed, not just the tip of main — otherwise a busy repo gives you phantom diffs from changes that merged while the PR was open.

The dbt PR Data Review Checklist

• •Blast radius: list modified models and everything downstream (state:modified+); flag changes to high-fan-out staging models.
• •Schema diff: any column added, removed, renamed, or retyped? Any contract or exposure affected?
• •Row counts: percent delta per modified model; zero expected for refactors; undefined (not zero) when the base is empty.
• •Column profiles: null rate, distinct count, min/max/mean on touched columns in both environments.
• •Value diff: PK-join classification (added / removed / changed / identical) for any model flagged above, with a bounded sample of changed rows.
• •Claim check: does the observed diff match what the PR description says should change? Mismatch = block.
• •Tests still meaningful: do existing tests cover the changed logic, or did the change route around them?
• •Cleanup: PR schema is disposable and gets dropped after merge.

When There Is No Primary Key

Some models legitimately lack a unique key — event grains, snapshot unions, intentionally exploded tables. Without a PK, rung 4 is unavailable in its precise form, and the honest move is to treat that as elevated uncertainty, not a pass. Fall back to a full-row hash comparison (hash the concatenation of all columns and diff the hash multisets) for an approximate changed-row count, lean harder on rung 3 profiles, and say plainly in the review summary that row-level attribution was not possible. A risk verdict that quietly skips the check it could not run is worse than no verdict at all. Long term, the absence of a testable grain is itself a finding worth a follow-up ticket.

Tooling

You can build the ladder yourself with the patterns above, and several tools package parts of it. The dbt-audit-helper package provides macros like compare_relations and compare_all_columns for in-warehouse comparisons. Recce wraps dbt PR review in a UI with lineage-aware diffs and checklists. Data Workers' Data Guardrail Agent runs this same ladder agentically — it builds the comparison, walks the rungs in cost order, and returns a machine-actionable risk verdict and summary that CI can gate on. See the product overview for how it fits alongside the rest of the swarm.

Reviewing a dbt PR without looking at the data is reviewing half the change. The ladder keeps the cost proportional to the risk: metadata first, counts second, profiles third, row-level truth only when the evidence demands it. More patterns like this live in our resources library.