You're on StreamNative: It Turns Your Pulsar and Kafka Streams Into Lakehouse Tables. Data Workers Owns Whether Those Tables Are Right
StreamNative streams Pulsar and Kafka events straight into Iceberg and Delta tables with Ursa. Data Workers catches a compatible schema change that breaks a model, traces it and gets the fix approved.
Your platform team runs StreamNative Cloud: Pulsar clusters with tenants, namespaces and topics, Kafka clients served through KSN or a Kafka cluster (Public Preview), all on the Ursa engine. Ursa writes every stream into Iceberg or Delta tables in the catalog you govern (Unity Catalog, Snowflake Horizon Catalog, S3 Tables or BigLake), with no connector in between. Since September 23, 2026 Ursa, Ursa for Kafka and the Lakestream specification are open source under Apache 2.0, and StreamNative is a founding member of the Streamhouse Working Group (September 15). Rio answers questions in the console, and the StreamNative MCP server brings your clusters into Claude Code or Cursor.
That design removes a pipeline and moves a responsibility. When a producer ships a schema that Pulsar accepts, the table evolves with it at compaction, and every dbt model, dashboard and forecast that reads it inherits the change. StreamNative keeps the schema valid. Whether the numbers built on it are still right is the job Data Workers does: it notices the number that moved, traces it to the change, and gets the fix approved by a named owner first.
Key takeaways
- •StreamNative keeps its job. Pulsar and Kafka on Ursa, schema compatibility and stream-to-table materialization stay with your platform team. Data Workers works on what the tables feed.
- •Compatible can still be wrong. A schema change that passes Pulsar's FULL check can empty a dbt join key. Data Workers catches the missing rows and names the change behind them.
- •It reads where the tables land. Snowflake, BigQuery, dbt and Azure Data Factory are native reads; Databricks tables and Iceberg REST catalogs come in over their APIs or MCP servers today. StreamNative's own MCP server sits next to Data Workers in your team's MCP client.
- •Every fix goes through a named person, who approves each proposal and makes the producer changes.
- •It stacks with Orca and Rio. StreamNative's agents operate the streaming estate; Data Workers runs everything the stream feeds, with approvals and receipts.
StreamNative turns your streams into lakehouse tables. Data Workers owns whether those tables are right.
The job downstream of StreamNative is to notice that a model built on a stream's table has gone wrong, find the change behind it, hold anything that acts on it, get the fix made where it lives and prove the result. Here is one Tuesday at a European EV charging network that buys its own electricity on the day-ahead market. This is an illustrative example.
The setup: StreamNative Cloud BYOC on Azure, a Latency-Optimized Pulsar cluster. Every charger backend produces to persistent://charging/sessions/session-ended with an Avro schema; the namespace uses FULL compatibility, Pulsar's default for Avro, and schema AutoUpdate is on, also the default. Ursa materializes the topic as a managed Iceberg table, charging.raw.session_ended, in Unity Catalog on Azure Databricks. An hourly Azure Data Factory pipeline, pl_charging_hourly, runs dbt on a Databricks SQL warehouse to build fct_charging_sessions. At 10:30 ADF starts the Databricks job day_ahead_load_forecast, and the trading desk submits tomorrow's power purchase before the exchange's 12:00 gate closure. The ops team opens the Power BI report "Network energy" at 09:30. The report and the forecast job are declared as dbt exposures and recorded in the context graph.
| Time | System | What happens |
|---|---|---|
| Tue 06:40 | Pulsar on StreamNative Cloud | The Nordic charger backend ships a release. Its producer brings Avro schema version 7: the optional station_id field (default null) is removed and an optional evse_id is added. FULL allows changes to optional fields, so AutoUpdate registers v7. The other regions keep producing v6 |
| 06:41 | Pulsar | Every subscription keeps reading. Consumers on v6 see station_id as null for Nordic sessions, which is what FULL promises |
| 06:50 | Ursa + Unity Catalog | Compaction materializes the new records. The table evolves with the source schema: a new evse_id column, and station_id null on every Nordic row. Nothing fails and nothing alerts |
| 07:10 | ADF + dbt on Databricks | pl_charging_hourly builds fct_charging_sessions, which inner-joins dim_stations on station_id. Nordic sessions, 38% of the network, drop out. The not_null and relationships tests pass, because every row that remains is valid |
| 07:12 | Data Workers | The pipeline's last step records sessions per region from the raw table and from the model, plus the station_id null rate, with monitor_metrics. Nordic sessions in the model read 0 against a baseline of about 2,300 for the hour, the raw table's Nordic count is normal, and Nordic station_id is 100% null. Data Workers opens an incident |
| 07:20 | Data Workers + dbt | The team's coding agent pulls the raw table's columns through Databricks' managed DBSQL MCP server, and Data Workers records evse_id added at 06:50. trace_cross_platform_lineage follows station_id from the raw table through stg_sessions into the join, using the dbt manifest. blast_radius_analysis returns the model, the Power BI report and day_ahead_load_forecast, whose same-hour features would read almost no Nordic load at 10:30. diagnose_incident names the cause: a new column and a null join key on Nordic rows since 06:50 |
| 07:25 | Microsoft Teams + email | A finding card lands in the charging data channel. The approval request goes by email to the data platform owner, with the Nordic backend lead copied. The platform engineer on call checks in her own Claude Code, through StreamNative's read-only MCP server: v7 dropped station_id |
| 08:05 | Spellbook | The owner signs in through Entra ID and reviews three proposals: hold the 10:30 forecast trigger until the model verifies; a dbt diff that resolves station_id from evse_id through dim_evse in stg_sessions; and a request to the Nordic backend team to send station_id again in v8. She approves all three |
| 08:10 | ADF | She stops the forecast trigger |
| 08:30 | GitHub | She merges the dbt diff |
| 08:35 | Data Workers + ADF | Data Workers queues the approved pl_charging_hourly rebuild for every hour since 06:00 with trigger_adf_pipeline; ADF finishes it at 08:58 |
| 09:00 | Pulsar | The Nordic team ships v8 with station_id back beside evse_id. AutoUpdate registers it |
| 09:15 | Data Workers | It verifies: Nordic sessions in the model match the raw table and sit on baseline for every hour since 06:00, and the owner reports the dbt tests on the rebuilt model passing. The receipt records the cause, the table change, approvals, runs, checks and the undo (revert the dbt diff; the raw table is untouched) |
| 09:20 | ADF | The owner re-enables the forecast trigger. The ops stand-up opens "Network energy" with every region in it |
| 10:30 | Databricks | day_ahead_load_forecast runs on full data, and the desk submits its bid at 11:40, ahead of gate closure |
Every part of StreamNative did its job: Pulsar applied the compatibility rule, AutoUpdate registered the schema, and Ursa evolved the table instead of failing the write. Knowing that station_id was the key a dbt join depends on, and that a forecast reads that join before a 12:00 auction, takes facts outside the stream.
| Job | What StreamNative does | What Data Workers does |
|---|---|---|
| The move | Streams every event over Pulsar or Kafka on Ursa, in your cloud account on BYOC | Reads the tables the stream lands in and every model and job built on them |
| The schema | Checks each new version against the namespace's compatibility strategy and registers it | Traces each field to every table, model, report and forecast that reads it |
| The tables | Materializes topics as Iceberg or Delta tables in your catalog and evolves them with the source schema | Detects the table change and compares region counts on the raw table with what the models built |
| The fix | Applies the producer, schema and namespace changes the owner makes, in the console, the CLI or its MCP server | Proposes the hold, the model change and the producer request to a named owner, then queues the approved downstream runs |
| The proof | Keeps cluster metrics and audit logs, and publishes Streaming Proof, an open harness for ordering and delivery guarantees | Re-checks the numbers and writes a receipt: what changed, who approved it, how it was checked, how to undo it |
Why doesn't StreamNative just do this itself?
Because StreamNative is built to make streams cheap, durable and queryable as tables, and it scopes its decisions to the stream. Accepting a FULL-compatible schema and evolving the table is the right behaviour for a streaming platform: producers keep shipping and no consumer breaks. Which dbt model joins on which field, and what a forecast does with the result, are facts StreamNative has no reason to hold.
StreamNative's AI work follows the same line. Rio, launched in the redesigned console on August 25, 2026 and in Public Preview, explores cluster health, configuration and resources; "every change goes through dry-run, approval, and diff review before it is applied." Orca Agent Engine, in Private Preview, is "the secure, governed runtime for AI agents" on Kafka and Pulsar events, with RBAC, audit and replay. The MCP server registers no write tools in read-only mode. All of it operates the streaming estate, which is where a streaming vendor should act.
Owning whether everything built on the tables is right, across Databricks, ADF, dbt and Power BI, is a different product with a different liability: a context graph, blast-radius scoping, named approvals, a recorded undo and receipts an auditor can read. That is Data Workers. See is it safe to let AI agents change production data.
Every tool owns a slice. Data Workers covers the whole lifecycle
Each point tool adds another console, contract and handoff. Data Workers covers the whole lifecycle with one context, one approval flow and one audit trail, and builds on the StreamNative streams already there.
| Stage | Data Workers | StreamNative | Why we scored it this way |
|---|---|---|---|
| Catalog & Context | 9 | 4 | Tenants, namespaces, topics and the lakehouse catalog describe the stream. Data Workers keeps one governed context graph from the table to the dbt model and the forecast job. |
| Analytics & Insights | 8 | 2 | SQL Workspace (Private Preview) queries topics in SQL. Data Workers answers data questions from governed definitions with lineage behind every number. |
| Data Quality | 8 | 3 | Pulsar's schema checks and the materializer's dead-letter path protect the table's shape. Data Workers checks what the rows mean to the models that read them. |
| Observability & Incidents | 8.5 | 5 | Cluster metrics, Rio (Public Preview) and Streaming Proof cover how the platform runs. Data Workers diagnoses the data incident across systems and verifies the fix. |
| Pipelines & Ingestion | 8.5 | 9 | StreamNative's home stage: native Pulsar and Kafka on Ursa, written straight into Iceberg and Delta tables. Data Workers plans the recovery and queues the downstream reruns. |
| Schema & Migration | 8 | 4 | Per-namespace compatibility strategies and table evolution keep schemas valid. Data Workers traces a change through every table, model and job that reads it. |
| Governance & Access | 8.5 | 6 | Multi-tenancy, RBAC and Orca's permissioned tool calls govern people and agents. Data Workers routes every data change to a named approver. |
| Security & Privacy | 8 | 6 | BYOC keeps data in your cloud account; Orca logs and replays agent actions. Data Workers leaves a receipt on every data change. |
| Cost / FinOps | 8 | 5 | Ursa's object-storage design and profiles cut streaming cost. Data Workers traces Snowflake spend to the dbt model and reads BigQuery and AWS spend. |
| MLOps & Models | 7.5 | 3 | Orca runs event-driven agents (Private Preview). Data Workers keeps the data under models, forecasts and agents healthy. |
How StreamNative and Data Workers work together
Data Workers connects where the stream lands. The tables Ursa writes are read natively in Snowflake and BigQuery; Databricks, Glue and Iceberg REST catalogs connect over their APIs or MCP servers today. dbt, Azure Data Factory, Microsoft Teams (alert cards) and Entra ID sign-in connect natively, among 50+ connectors; Power BI connects over its API today. Pulsar, Orca and StreamNative Cloud stay in StreamNative's console, CLI and MCP server, which your team's MCP client runs beside Data Workers.
By design, producer releases, schema uploads, compatibility settings, subscriptions and retention stay with the owner, in StreamNative's own tools. Data Workers proposes the steps, scopes what they touch and verifies afterwards. Backlog and throughput come from your monitoring; Datadog and OpenTelemetry are native reads.
Both MCP servers sit in the same client: StreamNative answers "what changed on the topic", Data Workers "what did it break, and what's the fix". StreamNative's open-source server, started with --read-only, registers only read tools; the remote server at mcp.streamnative.cloud (a preview feature) has a read-only access mode your admin sets. The Data Workers side follows the documented client setup: clone the open-source repo and add each agent's start-agent.sh entry.
# Example: StreamNative MCP server (read-only, schemas and topics) plus Data Workers agents in Claude Code
claude mcp add streamnative -- snmcp stdio --organization my-org \
--key-file /path/to/key-file.json --pulsar-instance my-instance --pulsar-cluster my-cluster \
--features pulsar-admin-schemas,pulsar-admin-topics --read-only
# Data Workers agents, from a clone of the open-source repo
claude mcp add --scope user dw-schema -- "$(pwd)/start-agent.sh" dw-schema
claude mcp add --scope user dw-catalog -- "$(pwd)/start-agent.sh" dw-context-catalog
claude mcp add --scope user dw-incidents -- "$(pwd)/start-agent.sh" dw-incidents
claude mcp add --scope user dw-connectors -- "$(pwd)/start-agent.sh" dw-connectorsList the tools with your client's own command (/mcp in Claude Code). In this incident: the team's coding agent reads the new column over Databricks' DBSQL MCP server and Data Workers records it; trace_cross_platform_lineage and blast_radius_analysis (dw-context-catalog) map the field to the join, the report and the forecast; monitor_metrics and diagnose_incident (dw-incidents) flag the regional gap and name the cause; trigger_adf_pipeline and send_teams_alert (dw-connectors) queue the approved rebuild and post the finding card; remediate re-checks the assertions and escalates any failure to a person.
In production the agents run in your infrastructure and hold the warehouse credentials and model key; your data stays in your systems, and the hosted Conductor sees workflow metadata only. See where does our data go.
One incident, L0 to L4, set per domain:
- •L0 manual. The desk pays imbalance prices for Nordic load it never bought. Someone finds the schema change by Thursday.
- •L1 observe. Data Workers flags the regional gap at 07:12 with the cause and blast radius. Nothing changes.
- •L2 propose. Data Workers proposes the hold, the model change and the producer request. Nothing changes until the named owner approves; an unanswered request expires and escalates.
- •L3 act reversibly. For a class with a clean record, Data Workers carries out reversible steps in the domain you open, such as queueing approved rebuilds. Producers and schemas stay with the owner.
- •L4 autonomous. For a scoped domain, producer schema files kept in Git reach Data Workers as pull requests to review. It checks each removed field against every model that reads it, so a dropped join key is flagged before the first record.
Who sets those levels: who owns the agents, how approvals work for AI data agents and autonomy levels L0 to L4 explained.
What changes for your team
- •Producers and consumers share one record. Platform, backend and analytics teams see one incident and receipt in Spellbook Data Catalog (in preview).
- •Schema changes get a downstream reviewer. Each change that lands in a table is read against the models that use its fields, by the Schema Evolution agent, so "compatible" also means "safe for the join".
- •Raw and modelled stay in step. Region and volume baselines on both sides of a model turn a silent gap into an incident with an owner, by the Incident Debugging agent.
See also the Streaming agent, real-time data pipelines for AI agents and what Data Workers adds on Databricks.
Keep StreamNative, or consolidate?
Keep StreamNative if you love it; Data Workers works with it from day one. Many teams consolidate once Data Workers runs that slice too.
For almost every StreamNative team the answer is keep it: Pulsar and Kafka on one engine, with streams that are already tables, is hard to give back. What teams consolidate is the tooling around the tables: a separate observability tool, hand-written row counts and "check the schema history" runbooks. Neighbours: you're on Redpanda, you're on Confluent and you're on Azure Data Factory; Data Workers integrations lists what connects natively, and what is an agentic data platform covers the category.
Building it yourself with a coding agent and the StreamNative MCP server? Read build it ourselves with Claude Code and MCP servers: reading a schema is easy; the context graph, approvals, undo and receipts are the work.
The case for your CFO
The outcome. When a stream change quietly removes a region or a segment from the tables behind forecasts, revenue or payouts, it is caught within the hour and fixed before money moves on it.
The risk story. At L1 agents only read. At L2 a named person approves each change; an unanswered request expires and escalates. L3 covers reversible changes in domains you open. Producers and schemas stay with the owner. No agent can promote its own work, an org-wide stop halts all autonomous dispatch, and every change carries a receipt with its blast radius and undo. Nothing migrates.
Why now. Ursa puts the stream straight into the tables finance, forecasting and agents read, with no pipeline in between to catch a change, so a schema change reaches a decision the same morning.
The first win. One stream that feeds money, such as a forecast or a billing run. L1 receipts show what would have been caught.
What stays the same. StreamNative, your producers, your catalog, Databricks, dbt, ADF and your on-call rota. For numbers, see the ROI of agentic data operations.
The sentence for upstairs: "StreamNative turns our streams into tables; Data Workers makes sure what those tables tell our forecasts and reports is right, and gets it fixed with our approval when it isn't."
Getting started
Start with a pilot. Pick the streams whose tables feed forecasts, billing or revenue, give Data Workers read access to the catalog schemas Ursa writes and the dbt project on top, and run at L1 for a few weeks. Then turn on L2 for one domain, and L3 for a narrow class once the receipts earn it. The pilot path is on the pricing page, and the pilot is credited in full against the first year.
FAQ
How does Data Workers connect to StreamNative? Where the stream lands: the Iceberg and Delta tables Ursa writes are read natively in Snowflake and BigQuery, along with dbt and your orchestrator; Databricks and Iceberg REST catalogs connect over their APIs or MCP servers today. StreamNative's MCP server runs in your team's own MCP client, next to Data Workers.
Why did a compatible Pulsar schema change break our model? For Avro and JSON, Pulsar's default strategy is FULL, which allows changes to optional fields, such as removing a field with a default, and AutoUpdate (on by default) registers any schema that passes. Old consumers read the removed field as its default, often null. The topic is fine; a dbt join on that field is not.
Will Data Workers change our Pulsar schemas, topics or namespaces? No. Producer releases, schema uploads, namespace policies and subscriptions stay with the owner. Data Workers catches the change where it lands, as a break in the table's metrics, and proposes the fix for the owner to make. Your engineers can read schema versions through StreamNative's MCP server, run with --read-only.
How does this relate to Orca and Rio? They do different jobs. Rio helps your team operate StreamNative Cloud, with approval and diff review before changes. Orca runs and governs event-driven agents on your streams. Data Workers runs everything downstream of the stream, with fixes approved by a named owner.
Ursa is open source now. Does that matter for Data Workers? It keeps the tables open. Streams land as standard Iceberg and Delta tables in your catalog, which is where Data Workers works, whichever engine wrote them.
Sources
- •StreamNative homepage, https://streamnative.io/ (checked Oct 3, 2026)
- •StreamNative docs, Cluster Profiles Overview (Latency-Optimized, Cost-Optimized; Kafka Clusters Public Preview), https://docs.streamnative.io/cloud/clusters/cluster-profiles-overview.md (checked Oct 3, 2026)
- •StreamNative docs, Lakestream Architecture, https://docs.streamnative.io/cloud/overview/lakestream-overview.md (checked Oct 3, 2026)
- •StreamNative docs, Catalog Preparation Guides, https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/README.md (checked Oct 3, 2026)
- •StreamNative docs, Unity Catalog Managed Iceberg Table on Azure, https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/azure (checked Oct 3, 2026)
- •StreamNative on Azure (Serverless and BYOC), https://streamnative.io/partners/azure (checked Oct 3, 2026)
- •StreamNative blog, "Open-Sourcing Ursa & UFK: Building an Open Ecosystem Around Stream Storage" (Sep 23, 2026), https://streamnative.io/blog/open-sourcing-ursa-ufk-building-an-open-ecosystem-around-stream-storage (checked Oct 3, 2026)
- •StreamNative blog, "Introducing the Stream Materialization Framework" (Sep 24, 2026), https://streamnative.io/blog/introducing-the-stream-materialization-framework-from-streams-to-governed-data-assets (checked Oct 3, 2026)
- •StreamNative blog, "StreamNative Joins the Streamhouse Working Group" (Sep 15, 2026), https://streamnative.io/blog/streamnative-joins-the-streamhouse-working-group-to-advance-an-open-architecture-for-real-time-applications-and-ai-agents (checked Oct 3, 2026)
- •StreamNative blog, "A New StreamNative Cloud Experience" (Aug 25, 2026; Rio), https://streamnative.io/blog/a-new-streamnative-cloud-experience-easier-navigation-faster-operations-and-ai-powered-assistance (checked Oct 3, 2026)
- •StreamNative blog index (Streaming Proof, Sep 25, 2026), https://streamnative.io/blog (checked Oct 3, 2026)
- •StreamNative, Rio (Public Preview; "Every change goes through dry-run, approval, and diff review before it is applied"), https://streamnative.io/products/rio (checked Oct 3, 2026)
- •StreamNative docs, SQL Workspaces (Private Preview), https://docs.streamnative.io/sql/overview/architecture.md (checked Oct 3, 2026)
- •StreamNative, Orca Agent Engine (Private Preview), https://streamnative.io/agent-engine/overview (checked Oct 3, 2026)
- •StreamNative MCP Server, README and Pulsar schema tools (v0.2.0, Aug 13, 2026), https://github.com/streamnative/streamnative-mcp-server (checked Oct 3, 2026)
- •StreamNative docs, Remote MCP Server and its access modes, https://docs.streamnative.io/agent-engine/sn-remote-mcp/remote-mcp-overview.md and https://docs.streamnative.io/agent-engine/sn-remote-mcp/remote-mcp-governance.md (checked Oct 3, 2026)
- •Apache Pulsar docs, Understand schema (FULL: "Modify optional fields"; FULL default for Avro and JSON; AutoUpdate on by default), https://pulsar.apache.org/docs/4.2.x/schema-understand/ (checked Oct 3, 2026)
- •Data Workers client setup guide, https://dataworkers.io/opensource-docs/client-setup/ (checked Oct 3, 2026)