What We Learned Studying the Data Engineering Market Before Building

Before we wrote a single line of product code, we spent four months doing something unsexy: reading earnings calls, mapping vendor acquisitions, interviewing data engineers, and building spreadsheets of market gaps. We are an early-stage startup, and we wanted to understand why the data engineering tooling market — despite billions in venture funding — still leaves teams firefighting at 2 AM.

Here is what we found.

The Market Is Consolidating Fast

The M&A activity in data infrastructure over the past three years tells a clear story. Databricks acquired Tabular for $1B+ (Iceberg). Snowflake bought Neeva (search/AI), Streamlit (apps), and Samooha (clean rooms). IBM acquired DataBand (observability). Informatica went private in a $8B deal. The pattern: every major platform wants to own the full stack.

This creates a problem for data teams. Vendor-neutral tooling is disappearing. If your data quality tool gets acquired by your warehouse competitor, your multi-cloud strategy just got complicated.

We decided early that Data Workers would be vendor-neutral. We sit on top of your existing stack — Snowflake, Databricks, BigQuery, Redshift, whatever you run — rather than replacing any piece of it.

The AI Agent Opportunity Is Real But Overhyped

Wes McKinney said on the Data Renegades Podcast that data infrastructure is "maybe one of the last frontiers of AI-resistant technology." He is right. Data engineering workflows involve ambiguity, institutional knowledge, and cascading consequences that make them genuinely hard for AI to handle autonomously.

And yet. The repetitive parts — triaging alerts, tracing lineage breaks, validating schema changes — are exactly where AI agents can save hours of toil. The opportunity is not "replace the data engineer." It is "give them back their weekends."

We identified 11 distinct agent types that map to real workflow gaps: incident debugging, quality monitoring, schema evolution, data context and cataloging, pipeline building, governance and security, cost savings and cleanup, data migration, data science and insights, and agent observability.

MCP Changes the Integration Math

The Model Context Protocol ecosystem now has 12,230+ servers. That number was in the hundreds a year ago. This matters because it means AI agents can connect to data tools without custom integration work for each one. We are building custom MCP servers, one per agent, to leverage this ecosystem.

What the Market Still Needs

Beyond the obvious (better observability, better quality), our research surfaced gaps nobody is filling well:

• •Migration automation. Every enterprise we talked to has a migration project — legacy on-prem to cloud, one warehouse to another, Hadoop decommissioning. These projects take 6-18 months and cost $2M-$5M. The tooling is mostly manual.
• •Cost optimization. Data teams know they are wasting money on unused tables and inefficient queries but lack tooling to systematically identify and clean up waste. Enterprises waste 30-40% of storage on data nobody queries.
• •Agent trust infrastructure. Only 13% of enterprises plan to deploy AI agents in production (Gartner). Not because agents do not work, but because there is no way to monitor and trust them.
• •Semantic grounding. When an AI agent writes a SQL query, it needs to know that "revenue" means "net recognized revenue, USD, excluding refunds" in your organization. Without a semantic layer, agent accuracy drops dramatically.

Where We Landed

We are building Data Workers as a swarm of 11 specialized AI agents for data engineering. Not a platform. Not a copilot. A coordinated set of agents that each do one thing well and work together through shared context.

We have working prototypes for our first agents and are in active design partner conversations with data teams. We are not pretending to have it all figured out. But four months of market research gave us conviction that the opportunity is real, the timing is right, and the existing vendors are not going to fill these gaps.

If you are a data engineer who has opinions about this, we want to talk to you.