You're on Greenplum: It Runs Your MPP Analytics. Data Workers Owns Whether the Data Behind It Is Right, in Place or Through a Move
Greenplum's public repo is archived and teams are weighing Broadcom, Cloudberry, WarehousePG, Greengage or a cloud move. Data Workers keeps the data right in place or through the move, behind approvals.
Your largest analytical tables live on Greenplum. The coordinator takes the query, the segments each hold a slice of every table by its distribution key, and the work runs in parallel across all of them. Nightly loads come in through gpfdist and external tables, append-optimized columnar tables hold years of history, and PostgresML, MADlib and pgvector run next to the data. Broadcom ships it as VMware Tanzu Greenplum; the 7.x line, based on PostgreSQL 12, reached 7.8.5 on September 15, 2026. Many DBAs have run it since the Pivotal days and know their gp_toolkit views by heart.
The platform question is open on a lot of these estates. The public Greenplum repository on GitHub has been archived since May 2024, and the community carried the code forward: Apache Cloudberry (incubating, 2.1.0 in April 2026), EDB WarehousePG and Greengage DB (7.5.1 in September 2026). Teams are weighing Broadcom's commercial Greenplum, a fork or a cloud warehouse. Either way, one job stays the same. Greenplum runs your MPP analytics. Data Workers owns whether the data behind it is right: it flags what each load recorded, traces a wrong number to its cause across every system around Greenplum, and hands the owner a fix and a run plan, whether a table stays or moves.
Key takeaways
- •Greenplum keeps its job. Segments, distribution keys, gpfdist loads and your DBAs' controls stay as they are.
- •A clean load gets checked too. Each load records the metrics your team defines and Data Workers flags the ones that drift, so an INSERT that succeeded with duplicate data becomes a diagnosed incident before anything downstream reads it.
- •Built around the engine, by design. Data Workers works from the metrics your loads record, the DAG runs in Airflow and the lineage in its context graph; your team's SQL stays the only SQL on the cluster.
- •Changes go to a named owner. DAG fixes arrive as a diff, cleanups as a statement the owner runs, reruns queue through your orchestrator after approval, and every step leaves a receipt.
- •The same loop runs a move. Whether you go to a fork or to Snowflake, Data Workers maps every reader, plans each wave with its parity checks and holds the completion gate for the owner's sign-off.
Greenplum runs your MPP analytics. Data Workers owns whether the data behind it is right.
Greenplum executes the SQL it is given across every segment, fast. Wrong data is a different job: notice it, connect it to a system Greenplum never sees, size what it reaches, fix the cause and prove the result. Here is one night at a regional electric utility with 1.1 million smart meters. Times are local. This is an illustrative example.
The stack: the meter data management system (MDMS) drops one interval read file per collector region on an ETL host each night. gpfdist serves that directory, and a readable external table, ext.ami_reads, points at gpfdist://etl01:8081/reads_*.csv. An Airflow 2 DAG, ami_interval_nightly, inserts the reads into ami.interval_reads, an append-optimized columnar table distributed by meter_id. Its last task, added during the pilot, runs three count queries the team wrote and records the results with Data Workers' monitor_metrics. A second DAG builds billing.daily_usage at 01:30; a 04:00 task exports billing determinants to the customer information system (CIS), and a 02:30 task copies the table to Snowflake in wave 2 of a reporting-mart move. Looker's "Load and usage" Explore reads both sides. The DAGs send OpenLineage events to Marquez; pages go through PagerDuty, approvals through Slack.
| Time | System | What happens |
|---|---|---|
| Mon 23:40 | MDMS | Collectors behind substations 14 and 22 come back after a communications outage. The MDMS writes Monday's reads for those 41,200 meters a second time, as reads_mon_s14s22_retry.csv, next to the regular file |
| Tue 00:30 | Airflow + gpfdist | ami_interval_nightly starts. The reads_*.csv pattern matches 13 files instead of 12 |
| 00:55 | Greenplum | The INSERT succeeds: 109.6 million rows instead of about 105.6 million. Every segment does its share; nothing fails |
| 01:05 | Airflow + Data Workers | The metric task records files loaded (13 against a baseline of 12), meters with more than 96 intervals for the day (41,200 against 0) and kWh for substations 14 and 22 (about twice their band). monitor_metrics flags all three |
| 01:08 | Data Workers | diagnose_incident reads the three signals together: one extra file, one extra day of intervals for exactly the meters behind the two substations that were offline. A duplicated batch, not a meter or tariff problem. get_incident_history finds no earlier case |
| 01:12 | Data Workers | blast_radius_analysis walks the context graph, where the team recorded the CIS export and the Snowflake copy, and the Explores Data Workers reads from Looker: ami.interval_reads feeds billing.daily_usage, the CIS export, the Snowflake copy and the Explore. 41,200 customers would be billed for double their Monday usage |
| 01:15 | PagerDuty + Slack | Data Workers pages the on-call data engineer with send_pagerduty_alert, recommends pausing billing_daily_usage and sends the approval request to the billing data owner in Slack, MDMS owner copied |
| 01:22 | Airflow | The on-call engineer pauses billing_daily_usage before its 01:30 run |
| 01:40 | Spellbook | The owner reviews the proposal: a diff to ami_interval_nightly that loads from the MDMS batch manifest instead of a file pattern, plus a guard task that stops the DAG when any meter has more than 96 intervals; a cleanup statement that deletes the duplicated rows from Monday's partition by load batch; and a run plan. The MDMS owner confirms the retry. She approves |
| 01:52 | GitHub + Greenplum | The data engineer merges the diff and runs the approved cleanup with psql under the ETL role |
| 02:00 | Airflow + Data Workers | The metric task runs again and records 12 files, 0 meters above 96 intervals and both substations inside their band. Nothing is flagged |
| 02:05 | Data Workers + Airflow | Data Workers queues the approved run of billing_daily_usage through Airflow with trigger_airflow_dag and reads its status. The on-call unpauses the schedule |
| 02:45 | Data Workers + Snowflake | The build and the Snowflake copy succeed. run_quality_check confirms the expected minimum row count on the copied table. The wave's parity query, run by the team, matches |
| 03:00 | Spellbook + PagerDuty | Data Workers writes the receipt (cause, approvals, diff, cleanup and who ran it, rerun, checks passed, undo) and resolves the page with resolve_pagerduty_alert. The approved fact "MDMS retries can resend a batch as a second file" lands in the Context Wizard graph |
| 04:00 | CIS + Looker | The billing export runs on schedule with single-counted usage, and the Explore shows both substations back in range |
Every system did its job: gpfdist served every file it was pointed at and Greenplum loaded every row. The wave 2 parity check would have passed too: both sides carried the same duplicates. Catching it took knowledge no engine was built to hold: a meter has 96 intervals a day, and a billing export reads that table before dawn.
| Job | What Greenplum does | What Data Workers does |
|---|---|---|
| The run | Loads through gpfdist and external tables, spreads each table across segments, answers queries in parallel | Reads the orchestrator's runs and the metrics each load records |
| The signal | gp_toolkit views, resource groups and segment status tell the DBA how the cluster is doing | Flags the team's load metrics against their baselines, so the owner knows whether the data is right |
| The diagnosis | Query plans and logs show what ran, where and for how long | Joins the signals, the DAG run and the lineage into one cause, with the blast radius across exports, copies and dashboards |
| The fix | Runs whatever SQL and loads the team deploys | Proposes the DAG change as a diff and the cleanup as a statement for the owner to approve and run |
| The rerun | Executes the jobs the scheduler sends | Queues the approved run through Airflow and reads its status |
| The proof | Logs the queries the cluster ran | Reads the re-run metrics, checks the Snowflake copy and writes a receipt: what changed, who approved it, how it was checked, how to undo it |
Why doesn't Greenplum just do this itself?
Because Greenplum is built to run very large analytical SQL in parallel across commodity hosts, and it does that well. Its unit is the query, the segment and the distribution key. Recent 7.x releases follow that job: faster index scans on append-optimized tables, optimizer plan hints, MADlib as an extension, and in-database machine learning through PostgresML, pgvector and PL/Container. Greenplum knows the INSERT committed 109.6 million rows; it has no reason to know a meter reads 96 times a day.
Owning whether a number is right across an MDMS, Airflow, Greenplum, a CIS export, a Snowflake copy and Looker is a different product: a context graph across every system, blast-radius scoping, approvals that name a person, a recorded undo and receipts an auditor can read. An engine that rewrote another team's DAGs would take on liability it was never designed to carry. And mid-decision, a team wants the operating layer to belong to no engine, so the checks, history and receipts carry over whatever it picks. That product is Data Workers. More in is it safe to let AI agents change production data.
Every tool owns a slice. Data Workers covers the whole lifecycle
Greenplum owns massively parallel analytics on its segments. Each point tool adds another console, contract and handoff. Data Workers covers the whole lifecycle with one context, one approval flow and one audit trail, and builds on the Greenplum estate already there.
| Stage | Data Workers | Greenplum | Why we scored it this way |
|---|---|---|---|
| Catalog & Context | 9 | 4 | The PostgreSQL catalog and gp_toolkit describe the tables, partitions and distribution policy inside the cluster. Data Workers keeps one governed context graph across Greenplum and every system around it. |
| Analytics & Insights | 8 | 8.5 | Greenplum's home stage: massively parallel SQL over segments, GPORCA and in-database analytics on very large tables. Data Workers answers data questions from governed definitions on top. |
| Data Quality | 8 | 4 | Constraints and the team's own SQL tests check what they are written to check. Data Workers flags the metrics that define "right" after every load and checks the tables that land in Snowflake. |
| Observability & Incidents | 8.5 | 3 | Segment status, resource groups and logs watch the cluster, not the business number. Data Workers diagnoses the data incident across the source, the orchestrator and the consumers, and verifies the fix. |
| Pipelines & Ingestion | 8.5 | 6 | gpfdist, external tables, PXF and the Streaming Server load data fast and in parallel; the schedule lives in Airflow or another orchestrator. Data Workers queues the approved rerun through the orchestrator. |
| Schema & Migration | 8 | 4 | Greenplum runs the schema it is given, and choosing where to go next is not its job. Data Workers plans moves in approved waves and generates Snowflake schema changes with rollback SQL for the owner. |
| Governance & Access | 8.5 | 5 | Roles, privileges and row-level security govern who reads what inside the cluster. Data Workers routes every data change to a named approver. |
| Security & Privacy | 8 | 6 | Postgres-grade security: roles, SSL, pg_hba rules and audit logging on the coordinator. Data Workers leaves a receipt on every data change across systems. |
| Cost / FinOps | 8 | 4 | Resource groups share a fixed on-premises cluster. Data Workers traces Snowflake credits to the dbt model through query tags on the side of the estate that moved. |
| MLOps & Models | 7.5 | 5 | PostgresML, MADlib, pgvector and PL/Container bring model training and scoring to the data. Data Workers keeps the data under those models healthy. |
How Greenplum and Data Workers work together
Data Workers works with Greenplum from the outside in, by design: it needs no seat on the coordinator. Your load DAGs run the team's own queries on Greenplum after each run and record the results with monitor_metrics, which flags any value that leaves its baseline. Airflow (DAG runs and task instances), Looker (read), Snowflake, PagerDuty, Slack and GitHub pull request review connect natively (Data Workers integrations). Your DBAs keep every grant, and the only SQL on the cluster is SQL your team wrote.
For ad hoc SQL, analysts and DBAs can run a community Postgres MCP server against Greenplum, in its read-only mode with a read-only role, next to Data Workers' agents in the same client. That server answers "show me the skew on this table"; Data Workers answers "what did last night's load do to the numbers, and what's the fix". See how to connect an AI agent to Postgres safely. The Data Workers side follows the documented client setup: clone the open-source repo and add each agent's start-agent.sh entry to your client.
// Example: Data Workers agents in one client config (Claude Desktop-style mcpServers block)
// Add your chosen Postgres MCP server for Greenplum alongside, in its read-only mode, per its README.
{
"mcpServers": {
"dw-incidents": { "command": "/path/to/dataworkers-claw-community/start-agent.sh", "args": ["dw-incidents"] },
"dw-catalog": { "command": "/path/to/dataworkers-claw-community/start-agent.sh", "args": ["dw-context-catalog"] },
"dw-quality": { "command": "/path/to/dataworkers-claw-community/start-agent.sh", "args": ["dw-quality"] },
"dw-schema": { "command": "/path/to/dataworkers-claw-community/start-agent.sh", "args": ["dw-schema"] },
"dw-connectors": { "command": "/path/to/dataworkers-claw-community/start-agent.sh", "args": ["dw-connectors"] }
}
}List the tools with your client's own command (for example /mcp in Claude Code). In this incident: monitor_metrics, diagnose_incident and get_incident_history (dw-incidents); blast_radius_analysis and trace_cross_platform_lineage (dw-context-catalog); run_quality_check on the Snowflake side (dw-quality); and trigger_airflow_dag, send_pagerduty_alert and resolve_pagerduty_alert (dw-connectors).
Through a move. If you move, the hard part is rarely the engine; it is the 400 readers. Data Workers' Data Migration agent plans each wave with its parity checks from the object inventory your team exports, maps who reads each table from the lineage in the native systems around Greenplum, proposes the comparison queries your team runs on both sides, tracks the results and holds the completion gate for the owner's sign-off. Both sides can match and still be wrong, so the load metrics keep running through every wave.
- •To a fork (Cloudberry, WarehousePG, Greengage). Each project points to its own migration path; Cloudberry names gpbackup. Data Workers' part is the reader map, the wave plan, the tracked parity checks, the signed gate and the same incident loop on the new cluster.
- •To Snowflake. Data Workers translates standard PostgreSQL-dialect SQL in your views and queries into Snowflake SQL for review, leaving low-confidence constructs annotated; Greenplum-specific clauses such as DISTRIBUTED BY stay with your engineers. On Snowflake it reads schemas, query history and row counts natively, runs quality checks after each load, and
generate_migrationwrites schema changes with rollback SQL for the owner to apply. See Data Workers on Snowflake.
Migration time of 4 to 8 weeks instead of 6 to 12 months is a design target. The agents run in your infrastructure with your credentials and model key; your data stays in your systems and the hosted Conductor sees workflow metadata only (where does our data go).
One incident, L0 to L4, set per domain:
- •L0 manual. Customers call on Wednesday about doubled bills, and billing spends the day matching files to rows.
- •L1 observe. Data Workers flags the duplicate day at 01:05 with cause and blast radius.
- •L2 propose. Data Workers proposes the DAG diff, the cleanup and the run plan. Nothing reaches production until the named owner approves; an unanswered request expires and escalates, never auto-grants.
- •L3 act reversibly. For a class with a clean record, Data Workers carries out reversible steps such as queuing the approved run, verifies them and sends any failed check to a person. Cleanups stay with the owner.
- •L4 autonomous. For a scoped domain, the fix is waiting for the owner before the first downstream DAG reads the table.
Who sets those levels: who owns the agents, how approvals work and autonomy levels L0 to L4 explained.
What changes for your team
- •The DBA stops being the first suspect. A wrong number from a clean load arrives with its upstream cause.
- •Exports that act on data get a guard. When a billing export or a regulatory extract would read a table that just went wrong, the hold is recommended before it runs.
- •The platform decision stops being urgent. The checks, receipts and context live outside the engine, so the engine choice can be made on its merits.
- •Everyone shares one record. Operations, platform and business owners see the same receipt in Spellbook Data Catalog (in preview), linked from the page.
Keep Greenplum, or consolidate?
Keep Greenplum if you love it; Data Workers works with it from day one. Many teams consolidate once Data Workers runs that slice too.
The engine is your platform decision, and Data Workers supports every answer: Broadcom's Greenplum, Cloudberry, WarehousePG, Greengage or a cloud warehouse. Teams that stay consolidate the tooling around Greenplum: check scripts copied into every load, a separate monitoring product, runbooks that say "reload and ask billing to look". Teams that move run the move through Data Workers. Neighbouring guides: you're on Teradata, you're on Postgres, you're on Airflow, you're on PagerDuty and energy and utilities.
Weighing a build on a Postgres MCP server and a coding agent? Running SQL is the easy part; see build it ourselves with Claude Code and MCP servers.
The case for your CFO
The outcome. When a Greenplum load sends a wrong number toward a bill, a regulatory report or a forecast, it is caught the same night and corrected before anyone acts on it, on Greenplum or mid-move.
The risk story. At L0 and L1, agents only read: the orchestrator, the lineage and the metrics your loads record. At L2 a named person approves every proposal; unanswered requests escalate, never auto-grant. At L3 agents act on reversible changes inside the domains you open; L4 is a later choice per domain. Cleanups stay with the owner. No agent can promote its own work, an org-wide stop halts all autonomous dispatch, and every change carries a receipt with the blast radius and the undo.
Why now. The open-source change put a platform decision on many Greenplum roadmaps, and a move doubles the places a number can go wrong. The operating layer you add now carries over to any engine.
The first win. L1 on the loads behind billing or regulatory reporting: a duplicated or missing batch becomes a diagnosed incident before anything downstream runs.
What stays the same. Greenplum, gpfdist loads, scheduler, BI tools and on-call rota. Zero migration to start; a move runs in approved waves. For the numbers, see the ROI of agentic data operations.
The sentence for upstairs: "Greenplum keeps running our big analytics while we decide its future; Data Workers makes sure the numbers are right and fixed with our approval, on Greenplum or wherever we move."
Getting started
Start with a pilot. Pick the Greenplum tables behind the numbers customers and regulators read, agree two or three metrics that define "right", record them from the loading DAGs with monitor_metrics, give Data Workers read access to the orchestrator and its lineage, and run at L1 for a few weeks. Then turn on L2 for one domain, and L3 for a narrow class once the receipts show the agents were right. The pilot path is on the pricing page, and the pilot is credited in full against the first year.
FAQ
Is Greenplum still open source? The public repository, now greenplum-db/gpdb-archive, has been archived since May 2024; Broadcom ships VMware Tanzu Greenplum commercially. Apache Cloudberry (incubating), EDB WarehousePG and Greengage DB are open-source projects built from the Greenplum code.
Which Greenplum open source alternative should we pick? That depends on licensing, support and how far you want to move. Cloudberry describes itself as built on a more modern PostgreSQL kernel with operational and behavioral compatibility; EDB describes WarehousePG as a community-driven alternative with high SQL parity; Greengage ships 6.x and 7.x lines. Data Workers works the same way with each and runs the move in approved waves.
How does Data Workers work with Greenplum? From the systems around it. Your DAGs record the metrics that define "right" after each load; Data Workers flags them, reads the DAG runs in Airflow, walks its context graph and checks the tables that land in Snowflake.
Will Data Workers write to our Greenplum cluster? No. DAG changes arrive as a diff for the owner to merge, data cleanups as a statement the owner approves and runs, and reruns queue through your orchestrator after approval. Your DBAs keep every grant.
Our parity checks pass. Why add load metrics during a migration? Parity proves the two sides match, not that either is right; a duplicated batch matches perfectly. Metrics such as intervals per meter, recorded with monitor_metrics on each load, catch what parity can't.
Sources
- •Broadcom, VMware Greenplum 7.x release notes (7.8.5 released Sep 15, 2026; PostgresML, pgvector, MADlib extension, PL/Container, plan hints, append-optimized index scans), https://techdocs.broadcom.com/us/en/vmware-tanzu/data-solutions/tanzu-greenplum/7/greenplum-database/relnotes-release-notes.html (checked Oct 3, 2026)
- •Broadcom, Greenplum 7 architecture overview ("based on PostgreSQL 12"; coordinator and segments), https://techdocs.broadcom.com/us/en/vmware-tanzu/data-solutions/tanzu-greenplum/7/greenplum-database/admin_guide-intro-arch_overview.html (checked Oct 3, 2026)
- •Broadcom, using the Greenplum parallel file server (gpfdist; wildcard patterns read every matching file), https://techdocs.broadcom.com/us/en/vmware-tanzu/data-solutions/tanzu-greenplum/7/greenplum-database/admin_guide-external-g-using-the-greenplum-parallel-file-server--gpfdist-.html (checked Oct 3, 2026)
- •Broadcom, monitoring a Greenplum system (gp_toolkit views, distribution checks), https://techdocs.broadcom.com/us/en/vmware-tanzu/data-solutions/tanzu-greenplum/7/greenplum-database/admin_guide-managing-monitor.html (checked Oct 3, 2026)
- •GitHub, greenplum-db/gpdb-archive (archived; last push May 23, 2024), https://github.com/greenplum-db/gpdb-archive (checked Oct 3, 2026)
- •Apache Cloudberry (Incubating), project site and FAQ, https://cloudberry.apache.org/ ; releases (2.1.0-incubating, Apr 15, 2026), https://github.com/apache/cloudberry/releases (checked Oct 3, 2026)
- •EDB, WarehousePG, https://www.enterprisedb.com/products/warehousepg ; https://github.com/warehouse-pg/warehouse-pg (checked Oct 3, 2026)
- •Greengage DB, repository and releases (7.5.1, Sep 25, 2026), https://github.com/GreengageDB/greengage (checked Oct 3, 2026)
- •Data Workers open-source repository (tool registrations in dw-incidents, dw-context-catalog, dw-quality, dw-schema, dw-connectors), https://github.com/DataWorkersProject/dataworkers-claw-community (checked Oct 3, 2026)
- •Data Workers client setup guide, https://dataworkers.io/opensource-docs/client-setup/ (checked Oct 3, 2026)