You're on Lightup: It Proves the Two Tables Disagree. Data Workers Makes Them Agree Again
Lightup checks data quality at scale and flags reconciliation gaps. Data Workers diagnoses across systems, fixes behind approvals, verifies and leaves a receipt.
Your data quality team runs on Lightup. Zero-config auto metrics cover freshness, volume and nulls on every table you connect, no-code checks and metric templates spread across thousands of columns, and prebuilt anomaly models learn what normal looks like, all pushed down to the source so the data stays where it is. During your migration, compare aggregate metrics track whether each target table still matches its source; release 4.26 (September 3, 2026) added IBM Informix as a data source and raises a separate incident per table pair. Lightup Hybrid runs the data plane in your own environment. Genie, Lightup's GenAI assistant in beta, now runs in agent mode by default, and Lightup Agentic exposes the platform as an MCP server for Claude Code and Gemini CLI. Lightup is the smoke alarm. Data Workers is the crew.
When a compare metric says the target is 1.8% short, someone still has to find why, fix the pipeline, reload the missing rows, rebuild the models and prove the report before the business reads it. Data Workers diagnoses, fixes and verifies that work behind approvals, and leaves a receipt.
Key takeaways
- •Lightup keeps its job. Auto metrics, checks, anomaly models, compare metrics, incidents, Genie and the Lightup Agentic MCP server stay where they are.
- •A reconciliation gap arrives with a cause and a fix. The Lightup incident reaches Data Workers through the on-call, whose assistant can use Lightup's MCP server side by side with Data Workers; Data Workers traces the gap through Dagster, Snowflake, dbt and Tableau, and proposes the pipeline change, the reload and the rebuild as one change set.
- •A named owner approves; Data Workers verifies. The owner approves in Spellbook, Data Workers queues the reruns through Dagster, then checks volume, quality and load lag against its baseline and confirms the compare metric reads zero.
- •Two MCP servers in one client. Lightup answers what disagrees; Data Workers answers why and fixes it.
- •Autonomy per domain. Each data domain climbs from L0 manual to L4 autonomous at the pace your team sets.
Lightup is the smoke alarm. Data Workers is the crew.
Lightup proves the two tables disagree. Data Workers makes them agree again and shows the receipt. Here is a Thursday at a regional grocery chain moving store sales from an on-premises IBM Informix consolidation database to Snowflake, across Informix, Dagster, Snowflake, dbt, Tableau and PagerDuty (times CDT). Lightup Hybrid runs in a Kubernetes cluster in the chain's data center and reads both sides. This is an illustrative example.
| Time | System | What happens |
|---|---|---|
| 01:40 | Informix | Nineteen Southwest stores whose network links were down Tuesday and Wednesday post two days of offline register sales, each line stamped with its original sale time |
| 02:15 | Dagster | The nightly store_sales_line asset reads Informix by a high-watermark on sale_ts and merges into RAW.POS.SALES_LINE in Snowflake; 212,400 back-dated lines sit behind the watermark and are skipped |
| 03:05 | dbt | Dagster materializes the dbt models; fct_store_sales_daily and two region models build on the short table |
| 04:30 | Lightup | The compare aggregate metric on row count and net sales, sliced by region, shows the Snowflake target 1.8% short for Southwest; Lightup raises an incident for the table pair and pages the data on-call through PagerDuty |
| 04:34 | Data Workers | Reads the incident and the compare metric over the Lightup API, reads the Dagster run, and finds the cause: the late lines carry sale times older than the watermark. Blast radius: three dbt models, the "Store P&L" Tableau workbook and the 08:00 regional ops call. Incident history shows the same pattern after an August outage. Data Workers annotates the PagerDuty incident with the cause |
| 04:50 | Data Workers | Proposes one change set: a diff to the Dagster asset that reads by the Informix posting time with a 72-hour lookback (the merge on sales_line_id keeps reloads free of duplicates), a rerun for the two-day window, a rebuild of the three dbt models, and a new dbt test that reconciles daily store counts against the landed lines |
| 05:35 | Spellbook | The sales data owner reviews the diff, the window and the blast radius, approves, and merges the change in the Dagster code repository |
| 05:40 | Dagster | Data Workers queues the window run and reads its status until it succeeds |
| 06:15 | Dagster and dbt | Data Workers queues the rebuild of the three models; the tests pass |
| 06:50 | Snowflake and Lightup | Data Workers verifies volume and quality on the raw table and all three models, and confirms the compare metric reads 0.0% for every region |
| 07:00 | PagerDuty | Data Workers resolves the PagerDuty incident with the receipt link |
| 07:30 | Tableau | The Store P&L extract refresh reads the corrected tables before the 08:00 call |
Lightup did its job: it read an on-premises source and a cloud target, compared them by region and raised the right incident at 04:30. What changed is everything after the alert: a cause traced to the pipeline upstream of the tables, a blast radius that reached the workbook and the meeting, one change set to approve, reruns queued in order, verification on every table the fix touched, and a record of what changed and how to undo it. Next time stores come back online with a backlog, the lookback catches the lines on the first run.
| Job | What Lightup does | What Data Workers does |
|---|---|---|
| Detection | Auto metrics, no-code checks, anomaly models and compare metrics catch volume, freshness, null and reconciliation gaps | Watches freshness and volume itself and catches schema changes in the dbt manifest and in review |
| Root cause | Shows which table pair, metric and slice disagree | Joins the incident to Dagster runs, dbt changes, lineage and incident history to find the cause and what else it touched |
| Routing | Sends the incident to PagerDuty, Slack, Jira or ServiceNow | Annotates the page with the cause and routes the fix approval to a named owner; unanswered requests expire and escalate |
| The fix | Data Remediation automates repair workflows from Lightup incidents, including scripts your team triggers from Airflow or other orchestrators | Proposes the complete repair (pipeline diff, reload window, dbt rebuild, new test) with blast radius as one change set |
| Running it | Your team runs the repair workflow or the pipeline | The owner approves and merges; Data Workers queues the reruns through Dagster |
| Verification | The compare metric clears on its next run | Checks volume and quality on every downstream table and confirms the compare metric |
| The record | The incident and its alert history | A receipt: the cause, the diff, who approved it, what it touched, how it was verified and how to undo it |
Why doesn't Lightup just do this itself?
Because Lightup made a clear, sensible choice about where its product ends. It checks data with pushdown queries, so it can watch thousands of tables without moving or touching the data. Genie changes Lightup's own configuration, enabling checks in bulk and creating monitors, and its docs say it "asks you to review the changes before applying them." The Lightup Agentic MCP server reads metrics, monitors and incidents and creates and updates metrics and monitors in plain English. Data Remediation automates the repair workflows a team builds around Lightup incidents.
That is the right line for a data quality platform. Lightup can be pointed at every table in an on-premises database and a cloud warehouse because nobody worries that a check will change a row, and a security team can approve a Hybrid data plane that only reads. Changing a Dagster asset, reloading a window of production data, rebuilding dbt models in Snowflake and proving a Tableau workbook is a different product with a different liability: check the change against everything downstream, get the owner's approval, have an undo before it runs, run the steps in order, prove the numbers and keep the record, in systems Lightup checks but does not run. Data Workers is built for exactly that job.
Every tool owns a slice. Data Workers covers the whole lifecycle
Lightup owns data quality checks and reconciliation at scale, and it is one of the strongest tools for proving a migrated table matches its source. Each point tool adds another console, another contract and another handoff. Data Workers covers the whole lifecycle with one context, one approval flow and one audit trail.
| Stage | Data Workers | Lightup | Why we scored it this way |
|---|---|---|---|
| Catalog & Context | 9 | 3 | Lightup has lineage and dashboards for the tables it checks and integrates with catalogs such as Alation and Collibra. Data Workers keeps one governed context graph of definitions, owners, lineage, quality and usage across every platform. |
| Analytics & Insights | 8 | 3 | Genie (beta) answers questions about checks and documentation. Data Workers answers data questions from governed definitions with lineage behind every number. |
| Data Quality | 8 | 8.5 | Lightup's home stage: no-code checks, zero-config auto metrics, templates and bulk creation, prebuilt anomaly models and compare metrics across table pairs, pushed down to the source. Data Workers writes and runs checks and proposes the dbt tests behind them. |
| Observability & Incidents | 8.5 | 8.5 | A shared home stage: incidents per metric and per table pair, routed to PagerDuty, Slack, ServiceNow and Jira. Data Workers diagnoses across systems, fixes with approval and verifies the fix. |
| Pipelines & Ingestion | 8.5 | 3 | Remediation workflows can be triggered from Airflow and other orchestrators. Data Workers queues reruns and backfills through your orchestrator, with approvals, and catches upstream changes first. |
| Schema & Migration | 8 | 5 | Compare metrics prove whether a migrated table matches its source. Data Workers catches schema changes, scores their blast radius and plans migrations in approved waves with parity checks. |
| Governance & Access | 8.5 | 2 | Workspaces and roles scope who configures which checks. Data Workers dry-runs each warehouse access request and proposes a time-bound grant for the owner to approve. |
| Security & Privacy | 8 | 2 | Lightup Hybrid keeps the data plane in your environment. Data Workers' pull request review flags new columns whose names or annotations look sensitive, and leaves a receipt on every data change. |
| Cost / FinOps | 8 | 2 | Pushdown queries keep check costs in the source. Data Workers attributes Snowflake spend to dbt models and drafts warehouse setting changes for their owner. |
| MLOps & Models | 7.5 | 2 | Lightup checks the tables under models and reports. Data Workers keeps the data under your models fresh and correct. |
Lightup leads on data quality and ties on incidents. For the category view, read how Data Workers is different from data observability and Bigeye, Anomalo, Sifflet and Soda vs Data Workers. For a direct comparison, see Data Workers vs Lightup and beyond data observability: autonomous resolution.
How Lightup and Data Workers work together
Lightup stays on top, where checks fire, compare metrics flag gaps and Genie helps the team tune coverage. Spellbook Data Catalog (in preview) is where the data team reviews each proposed change and how to roll it back. Between them, Data Context Wizard keeps one governed context graph, the Data-Agents Swarm does the work with 20 specialized agents, and the Autonomous Data-Conductor runs each fix end to end: detect, diagnose, fix, review, verify, remember.
Lightup to Data Workers. Lightup connects over its API or MCP server today: your team's assistant reads incidents, metrics, compare metrics and monitors side by side with Data Workers, whose own checks rest on Dagster, Snowflake and dbt. Snowflake, Dagster, dbt, PagerDuty and Tableau (read by design) are native connectors. Informix stays where Lightup Hybrid reads it, so Lightup's compare metric is the source-side evidence. A new Lightup incident starts a diagnosis: diagnose_incident and get_root_cause work the evidence, trace_cross_platform_lineage and blast_radius_analysis map everything downstream, the dbt manifest rules out a model change, and get_incident_history checks for repeats. remediate proposes the change set; the pipeline change arrives as a diff for the owner to merge, and reruns are queued through Dagster. run_quality_check and get_quality_score verify, and get_audit_trail reads back every step. Checks, compare metrics and Genie settings stay with your Lightup admins.
Side by side in one client. Lightup's Claude Code plugin finds the API credential file you download from your Lightup profile. Every Data Workers agent is an MCP server; the client setup guide documents the path: clone the open-source repo and add one start-agent.sh entry per agent. Example for Claude Code, with the Lightup commands as its repository documents them:
# Lightup Agentic (plugin finds lightup-api-credential.json)
claude plugin marketplace add lightup-data/lightup
claude plugin install lightup-ai@lightup
# Data Workers agents
claude mcp add dw-incidents -- /path/to/dataworkers-claw-community/start-agent.sh dw-incidents
claude mcp add dw-context-catalog -- /path/to/dataworkers-claw-community/start-agent.sh dw-context-catalog
claude mcp add dw-quality -- /path/to/dataworkers-claw-community/start-agent.sh dw-quality
claude mcp listAsk "why is Southwest short in the store sales reconciliation, and what does a fix touch?" and the client calls both: Lightup returns the incident, the compare metric and the slice; Data Workers returns the watermark cause, everything downstream and a proposed repair. Lightup's credential scopes what the client sees in Lightup; Data Workers' guardrail decides whether a change may run in that domain. For a shared endpoint, the Data Workers remote server takes an API key (bearer) or OAuth tokens from your identity provider, such as Okta, verified through JWKS.
One Lightup incident, L0 to L4. The same reconciliation gap at each level of the autonomy ladder, set per domain.
- •L0 manual. An engineer reads the incident, digs through Dagster runs, edits the asset and reruns everything by hand.
- •L1 observe. Data Workers posts the cause and blast radius. Nothing changes.
- •L2 propose. Data Workers proposes the diff, the reload window, the rebuild and the new test; nothing runs until the owner approves.
- •L3 act reversibly. For proven change classes, such as rerunning a merged asset for a bounded window, Data Workers queues the step through Dagster, verifies it and records the receipt.
- •L4 autonomous. In a scoped domain with a long clean record, Data Workers handles the repeatable steps end to end; pipeline changes still go to their owner as a diff.
For the safety model, read is it safe to let AI agents change production data, how approvals work and autonomy levels L0 to L4. On where data lives: the agents run in your infrastructure, your data stays in your systems, and the hosted Conductor sees workflow metadata only. That pairs naturally with Lightup Hybrid.
What changes for your team
Lightup made every reconciliation gap visible by table pair and slice. Data Workers gives the owner a crew.
- •Incidents. A Lightup incident arrives with a diagnosis, a blast radius and a proposed fix, and closes with a receipt.
- •Data quality. Lightup keeps checking at scale; each fix adds the dbt test that catches the repeat upstream.
- •Cloud spend. Snowflake spend is attributed to dbt models, and warehouse setting changes go to their owner drafted.
- •Access. A warehouse access request is dry-run and becomes a scoped, time-boxed grant proposed for the data owner.
- •Audits. Lightup records the incident; Data Workers records what changed, who approved it and how to undo it.
- •Migrations. Each wave is planned with its parity checks, tracked, and held at the completion gate for the owner's sign-off, with Lightup's compare metrics as the evidence.
The migration team changes most: a compare metric that goes red at 04:30 lands on a ready change set before the morning stand-up. See the data incident response playbook, data migration automation with AI agents and who owns the agents.
Keep Lightup, or consolidate?
Keep Lightup if you love it; Data Workers works with it from day one. Many teams consolidate once Data Workers runs that slice too.
Teams usually keep Lightup where it checks at the widest scale: thousands of tables across on-premises databases and cloud warehouses, and reconciliation through a long migration. What they consolidate is the stack around it: a second monitoring tool alerting on the same gap, hand-written repair scripts, ad hoc rerun runbooks. Weighing a fix layer of your own on the Lightup MCP server and a coding agent? Read build it ourselves with Claude Code and MCP servers: the MCP calls are the easy part; the context graph, approvals and rollback are the work. Across the category, see you're on Monte Carlo, you're on DQLabs and you're on Sifflet, plus what is an agentic data platform and, for this stack, Data Workers on Snowflake.
The case for your CFO
The outcome: you already pay Lightup to prove your data is right, table by table and through every migration wave. Data Workers turns each gap it finds from a morning of tracing, editing and rerunning into a diagnosis on arrival and a complete fix with one approval, so store operations, finance and merchandising read correct numbers at their first meeting, and migration waves close on evidence.
The risk story is plain. Autonomy is set per domain: L1 only reads; L2 changes nothing until a named person approves; L3 applies changes it can undo. Pipeline and model changes always go to their owner as a diff to merge. An unanswered approval request expires and escalates, never auto-grants. No agent can promote its own work. Every change carries a receipt: the cause, the diff, who approved it, what it touched, how it was verified and how to undo it. An org-wide stop halts all autonomous dispatch. Zero migration: Lightup, Informix, Dagster, Snowflake, dbt, Tableau and PagerDuty stay where they are.
Why now: Lightup's agent features and MCP server make the detection side conversational, so the slow part is the change, the reload and the proof. The first win is read-only: every Lightup incident in one domain gets a cause and a blast radius, while checks, permissions and pipeline review stay the same. For the numbers, see the ROI of agentic data operations.
The sentence to repeat upstairs: "Lightup proves where our data disagrees; Data Workers makes it agree again with one approval and shows us the receipt."
Getting started
Start with a pilot. Pick one domain whose checks already live in Lightup, such as store sales during the migration, connect Data Workers to Lightup, Dagster, Snowflake and dbt, and run at L1 so every incident gets a cause and a blast radius. Then turn on the first write class at L2, such as windowed reruns after a merged fix, with owner approval. The pilot path and plans are on the pricing page, and the pilot is credited in full against the first year.
FAQ
How does Data Workers connect to Lightup? Over Lightup's API or MCP server today, reading incidents, metrics, compare metrics and monitors. You can also run the Lightup Agentic plugin beside the Data Workers agents in Claude Code.
Lightup already has Data Remediation and Genie. Why add Data Workers? Genie changes Lightup's own configuration after you confirm, and Data Remediation automates repair workflows your team builds from Lightup incidents. Data Workers adds the cross-system work: the diagnosis in the pipeline that caused the gap, the change set with its blast radius, a named approver, reruns queued in order, verification on every downstream table and the receipt.
Does Data Workers change anything in Lightup? No. Checks, compare metrics, monitors and Genie settings stay with your Lightup admins. The receipt lives in Spellbook and the audit trail, linked from the PagerDuty incident.
We run Lightup Hybrid so data never leaves our environment. Does Data Workers fit that? Yes. The agents run in your infrastructure and hold the warehouse credentials, so your data stays in your systems. The hosted Conductor sees workflow metadata only: goals, table names, proposals with diffs, run records, approval handles.
Does Data Workers edit our pipelines or delete rows directly? Pipeline and model changes arrive as a diff for the owner to merge. Reloads run through your existing pipelines after approval, and any data cleanup is proposed for the owner to approve and apply. Data Workers queues the reruns through your orchestrator and verifies the result.
Can Data Workers help run the migration Lightup is reconciling? Yes. Data Workers plans each migration wave with its parity checks, tracks them and holds the completion gate for the owner's sign-off. Lightup's compare metrics become the evidence each wave closes on.
Sources
- •Lightup, homepage (Agent (Beta), Genie (Beta); © 2026), https://lightup.ai/ (checked Oct 3, 2026)
- •Lightup Docs, release notes (4.26, 03/09/2026 = Sep 3, 2026: IBM Informix data source, compare aggregate metric incidents per table pair, metric templates and bulk creation, Genie agent mode on by default; 4.25, Dec 16, 2025: Genie creates monitors after confirmation), https://docs.lightup.ai/docs/release-notes.md (checked Oct 3, 2026)
- •Lightup Docs, Genie: Lightup's GenAI assistant (Beta), https://docs.lightup.ai/docs/genie-ai-assistant-beta.md (checked Oct 3, 2026)
- •Lightup Docs, compare aggregate metrics, https://docs.lightup.ai/docs/compare-aggregate-metrics.md (checked Oct 3, 2026)
- •Lightup Docs, deployment models (Cloud and Hybrid; Hybrid data plane in a VM or Kubernetes cluster in your environment), https://docs.lightup.ai/docs/deployment-models.md (checked Oct 3, 2026)
- •Lightup Docs, PagerDuty integration, https://docs.lightup.ai/docs/pagerduty.md (checked Oct 3, 2026)
- •Lightup Docs, ServiceNow integration, https://docs.lightup.ai/docs/servicenow.md (checked Oct 3, 2026)
- •Lightup, Data Remediation, https://lightup.ai/data-remediation/ (checked Oct 3, 2026)
- •Lightup Agentic MCP server repository (Claude Code plugin, Gemini CLI extension; README lists 40 tools across metrics, monitors, incidents, datasources, integrations and docs), https://github.com/lightup-data/lightup (checked Oct 3, 2026)
- •Data Workers open-source repository, https://github.com/DataWorkersProject/dataworkers-claw-community (checked Oct 3, 2026)
- •Data Workers client setup guide, https://dataworkers.io/opensource-docs/client-setup/ (checked Oct 3, 2026)