Quality Agent Anomaly Detection

Data Workers' Quality Agent performs continuous anomaly detection across data pipelines, identifying volume spikes, distribution shifts, freshness violations, and schema drift before they propagate to downstream consumers. Unlike static threshold alerts, the agent uses statistical models that learn normal patterns and adapt to seasonality, making it effective on datasets where static rules generate constant false positives.

This guide covers the Quality Agent's anomaly detection algorithms, signal types, integration with alerting platforms, and strategies for tuning sensitivity across datasets with different variability profiles.

Why Statistical Anomaly Detection Beats Static Thresholds

Static threshold alerts are the most common data quality monitoring approach and the least effective. A rule like 'alert if row count drops below 900,000' works until the business grows and the normal row count reaches 1,200,000 — at which point a 25% drop to 900,000 goes undetected. Worse, the same static threshold generates false positives during known low-volume periods like holidays.

The Quality Agent replaces static thresholds with statistical models that learn what normal looks like for each dataset. It builds baseline distributions from historical data, decomposes seasonal patterns (daily, weekly, monthly), and sets dynamic thresholds that move with the data. A 10% drop that would be noise on a high-variance table triggers an alert on a table that normally varies by less than 1%.

Detection Algorithms

The Quality Agent employs a layered detection approach. For time-series signals (row counts, freshness, latency), it uses STL decomposition to separate trend, seasonal, and residual components, then applies modified Z-score detection on the residual. For distribution signals (column value distributions), it uses KL divergence to measure how much today's distribution differs from the baseline. For categorical signals (new enum values, missing categories), it maintains expected value sets and flags deviations.

The multi-algorithm approach is essential because no single algorithm works for all data patterns. Row counts follow time-series patterns. Value distributions are best compared with statistical divergence measures. Categorical columns need set-based comparison. The agent automatically selects the appropriate algorithm for each column based on the data type and observed variability.

• •STL decomposition — separates seasonal and trend components for clean anomaly detection on time-series metrics
• •Modified Z-score — robust outlier detection using median absolute deviation, resistant to masking by extreme values
• •KL divergence — measures distribution shift between current and baseline value distributions
• •Isolation Forest — detects multivariate anomalies across correlated columns simultaneously
• •Prophet-based forecasting — generates expected value ranges with uncertainty bands for business metrics
• •Set difference detection — identifies unexpected new values or missing expected values in categorical columns

Signal Types and Coverage

The Quality Agent monitors five signal categories across every table in the data warehouse. Volume signals track row counts and byte sizes. Freshness signals track data arrival timing and processing latency. Distribution signals track column-level value distributions. Schema signals track structural changes. Relationship signals track referential integrity and cross-table consistency.

Full coverage means that an anomaly anywhere in the data pipeline is detected within minutes. A source system that stops sending data (freshness), a ETL bug that doubles records (volume), a code change that shifts currency values from dollars to cents (distribution), a migration that renames a column (schema), or a broken join that produces null foreign keys (relationship) — the agent catches all five without any custom rule configuration.

Adaptive Baselines and Seasonality

Real-world data has multiple overlapping seasonal patterns. E-commerce data peaks on weekends. Financial data follows monthly reporting cycles. Advertising data spikes during campaigns. The Quality Agent decomposes these patterns automatically and builds baselines that account for all observed seasonality. Anomalies are only flagged when the data deviates from its expected seasonal pattern, not from a flat average.

Baselines adapt continuously. As the business grows, the agent's model of normal grows with it. When a product launch permanently increases data volume, the agent treats the initial spike as an anomaly (correctly), then updates the baseline to reflect the new normal within a few days. This adaptive approach eliminates the threshold maintenance burden that makes static monitoring unsustainable at scale.

Alerting and Integration

Detected anomalies are classified by severity based on the magnitude of deviation and the criticality of the affected table. Critical anomalies on tier-1 tables trigger immediate PagerDuty alerts. Warning-level anomalies on non-critical tables create Slack messages in the data quality channel. All anomalies are logged in the quality dashboard for trend analysis and reporting.

The Quality Agent integrates with the Incidents Agent for root cause correlation. When an anomaly is detected, the Incidents Agent checks whether a known upstream issue (schema change, pipeline failure, infrastructure event) explains the anomaly. If a root cause is found, the anomaly alert is enriched with the explanation, preventing the data team from investigating an issue that is already being resolved.

Tuning Sensitivity per Dataset

Different datasets require different sensitivity levels. A financial ledger table that should be perfectly consistent needs tight thresholds. A web analytics table with high natural variance needs relaxed thresholds. The agent supports per-table and per-column sensitivity configuration, and provides a tuning wizard that analyzes historical alert patterns and recommends adjustments to minimize false positives while maintaining detection of real issues.

For teams building comprehensive data quality programs, anomaly detection works alongside Great Expectations generation to provide both rule-based testing and statistical monitoring. Rule-based tests catch known failure modes; anomaly detection catches unknown unknowns. Together, they provide defense in depth for data quality. Book a demo to see anomaly detection running on your data warehouse.

Statistical anomaly detection eliminates the false positive flood that kills static threshold monitoring. The Quality Agent learns what normal looks like for every table, adapts to seasonality and growth, and alerts only when real anomalies occur — making data quality monitoring sustainable at warehouse scale.