Data Quality Auditor by Hand vs With an AI Agent

A data quality audit done by hand means a data engineer profiles a dataset, writes checks in a test framework such as Great Expectations or dbt data tests, runs ad hoc SQL for everything those checks miss, and writes the findings up as a report. An AI agent can run that same sequence end to end: our dw-data-quality-auditor skill audits a dataset and produces a findings report across all quality dimensions. A person still decides what the findings mean and what to change.

Key Takeaways

• •A manual audit is mostly coordination work: profiling, rule-writing, ad hoc SQL and report-writing happen in different tools, and the engineer is the glue between them.
• •The dw-data-quality-auditor skill, run by our Quality Agent, audits a dataset end to end and returns one findings report across all quality dimensions.
• •An agent changes who does the gathering and drafting. It does not change who owns the decision about what counts as acceptable quality for a given table.
• •Fixes, rule changes and anything that writes to production data should stay behind human approval.
• •Evaluate an audit agent on a dataset where you already know the defects: check whether it finds them, whether it invents any, and whether every finding links back to evidence.

Data quality auditor by hand vs with an AI agent: the decision in brief

The choice is not between rigor and speed. A careful manual audit is rigorous, and we are not arguing otherwise. The real difference is repeatability. A manual audit happens when someone has the time, covers the dimensions that person remembers to check, and goes stale the day the schema changes. An agent audit is a procedure that runs the same way each time and can be re-run on request.

If you audit one dataset a year, doing it by hand is reasonable. If you audit many datasets, or re-audit after every significant change, the labor in the manual path is what you are deciding about. We do not have a published benchmark for audit time, so we describe effort qualitatively throughout this post rather than quote numbers we cannot source.

The job and who does it today

The job is to answer one question about a dataset: can the people who depend on it trust it, and where specifically can they not? The answer needs to cover more than null counts. Teams usually think in terms of completeness, uniqueness, validity, consistency, accuracy and timeliness, and a full audit touches all of them.

Today the work usually lands on an analytics engineer or data engineer, often after a complaint from a stakeholder, an incident, or a migration. Sometimes a governance or compliance team requests it ahead of an external review. In every case the auditor needs access to the warehouse, knowledge of how the table is meant to behave, and enough context to tell a real defect from a quirk of the business.

The manual workflow, step by step

Here is the sequence as we see it in most teams. The tools vary; the shape does not.

• •Scope the dataset. Agree which tables are in, which consumers matter, and what "good" means for each. This is a conversation, not a query, and it often happens in chat threads.
• •Profile the data. Run row counts, null rates, distinct counts, min/max and value distributions, usually with a profiling notebook or a handful of SQL queries.
• •Write checks. Encode expectations as tests in a framework such as Great Expectations or dbt tests: not-null, unique, accepted values, relationships between tables.
• •Run ad hoc SQL for what the framework does not cover: duplicate near-matches, impossible dates, joins that silently drop rows, stale partitions.
• •Check freshness and volume against what the pipeline is supposed to deliver, which often means opening the orchestrator in another tab.
• •Triage. Separate real defects from expected behavior by asking the people who own the source system.
• •Write the report. Collect findings, evidence queries, severity and proposed fixes into a document someone else can act on.

Where the manual audit breaks

The failure points are consistent. Each step lives in a different tool, so context is lost at every handoff: the profiling notebook does not know which tests exist, and the test results do not know what the stakeholder asked. The auditor carries all of it in their head.

Coverage is the second problem. Rule-based test frameworks check what someone thought to write down. A column that drifts slowly, or a join that starts dropping a few rows after a migration, passes every rule that was written before the change. Finding those requires going beyond the existing tests, and that is the step most often cut when time is short.

The third problem is that the report is a snapshot. It is accurate on the day it is written. Rerunning it means repeating most of the work, so in practice it is not rerun until the next crisis.

What an agent does instead

The dw-data-quality-auditor skill belongs to our Quality Agent, which monitors data quality by wrapping Great Expectations, dbt tests and anomaly detection. The skill's job is exactly what its description says: audit a dataset end to end and produce a findings report across all quality dimensions. We are deliberately not claiming more than that.

In practice that means the steps above run as one procedure instead of seven separate chores. Because every Data Workers agent is an MCP server, you invoke it from the tool you already work in, such as Claude Code or Cursor, rather than opening a separate platform. The Model Context Protocol is the open standard that makes this possible: any MCP-compatible client can call the same tools.

A request looks like the questions our users already type: "Why did data quality score drop below SLA?" or "Show freshness scores for all gold tables." The audit skill is the heavier version of those questions, run across the whole dataset with the output collected in a single report.

The Quality Agent also chains with other agents when a finding needs context. Our Catalog Agent) supplies what a column is supposed to mean, and our Schema Agent can map the blast radius when a finding points at a column that other tables depend on. We covered how those agents share context in a separate post on why a coordinated agent swarm beats point tools.

Manual audit vs agent audit

What still needs a human

An audit finds problems. Deciding what to do about them is a different job, and we keep it with people. The open-source edition of Data Workers, dw-claw, is read-only by default, which suits an audit: the skill can inspect without changing anything.

• •Defining acceptable quality. Whether a 2% null rate in a column is a defect or normal depends on the business. An agent can report the rate; the owner sets the threshold.
• •Changing tests or rules. Adding, loosening or removing a check changes what the team is alerted about. Review it like any other code change.
• •Remediation. Backfills, deletes and corrections write to production data. In Pro, writes run through approval workflows with audit trails, and we recommend keeping them that way for anything a consumer relies on.
• •Sensitive data. If the audit surfaces PII in an unexpected place, that is a governance decision. Our Governance Agent handles PII detection and audit trails, but a person approves policy changes.
• •Sign-off. If the audit supports a compliance review, a named human signs the report.

How to evaluate an agent for this job

Do not judge an audit agent by how confident its report sounds. Judge it by a test you control. We suggest this sequence, and it applies to any audit tool, ours included.

• •Pick a dataset where you already know the defects. Plant a few if you have to: duplicate keys, an out-of-range date, a stale partition, a join that drops rows.
• •Run the agent and count how many known defects it found, and how many it missed.
• •Check for invented findings. Every item in the report should point to evidence you can re-run, such as a query or a test result.
• •Check dimension coverage. Ask whether the report addresses every quality dimension you care about, or only the easy ones.
• •Re-run it after you change the data. A useful audit changes when the data changes.
• •Confirm what it can write. For a first trial, use a read-only setup and verify nothing changed.

This is the same discipline you would apply to a new teammate's first audit. We would rather you test the skill on your own known-bad data than take our word for it.

Frequently Asked Questions

What is a data quality auditor? It is the role, or the task, of examining a dataset against defined quality dimensions and reporting what is wrong, where, and how badly. Historically a person does it with profiling queries and a test framework; an agent skill can now run the same sequence.

Can an AI agent replace a data quality auditor? Not the judgment. The dw-data-quality-auditor skill takes over the gathering, checking and report-drafting, but a person still sets thresholds, confirms which findings are real and approves any fix that writes to production data.

What does the dw-data-quality-auditor skill produce? A findings report across all quality dimensions for the dataset you point it at. It is run by our Quality Agent, which wraps Great Expectations, dbt tests and anomaly detection.

Do I need to stop using Great Expectations or dbt tests? No. Those frameworks encode the rules your team has already written, and the Quality Agent builds on them instead of replacing them. The agent adds the coordination and the checks nobody wrote.

Does the audit change my data? Not in the open-source edition, which is read-only by default. Writes belong to Pro, where they go through approval workflows and audit trails.

How do I run it from Claude Code or Cursor? Every Data Workers agent is an MCP server, so you connect it to any MCP-compatible client and invoke the skill from your terminal or editor. Start from the open-source edition so the first run is read-only.