You're on Elastic: Kibana Fires the Alert. Data Workers Fixes the Data Behind It
Elastic holds your logs and your RAG index, and Agent Builder investigates. Data Workers traces a stale index to the broken model, fixes it with approval and proves it.
Your company already runs on Elastic. Pipeline and application logs land in data streams, Streams parses them on the way in, and Kibana rules written in ES|QL raise alerts that open cases. The SOC works in Elastic Security. The search team runs Elasticsearch as the retrieval layer for a RAG assistant, with semantic_text fields and hybrid search over the product catalog. Elastic Agent Builder has been generally available since Elastic Stack 9.3, and since 9.4 it is the default chat experience. Elastic is where logs, search and observability data live and alerts fire. Data Workers is where data incidents get diagnosed, fixed and verified.
Elastic will tell you the shop assistant started failing at 09:20 and the newest document in the index is 31 hours old. The cause sits upstream, in a dbt model on Databricks that failed a test overnight while every job reported success. Data Workers finds it, repairs it with a named approver, checks the numbers and keeps a receipt in Spellbook that the case links to.
Key takeaways
- •Elastic keeps its job. Streams, Kibana rules and cases, Elastic Security and Agent Builder stay where they are.
- •A green pipeline can feed a stale index. When a dbt model is skipped, the reindex job still succeeds on yesterday's rows. Data Workers traces the stale index back to the model that broke.
- •Two MCP servers, one client. Run Data Workers beside the Elastic Agent Builder MCP server: Elastic answers what fired and what the index holds, Data Workers fixes what broke in the data.
- •Elasticsearch stays your vector store. Data Workers keeps the tables that feed the RAG index correct and fresh; retrieval, embeddings and ranking stay in Elastic.
- •Autonomy per domain. Each domain climbs from L0 manual to L4 autonomous at your pace, with approvals, rollback and receipts.
Elastic is the smoke alarm. Data Workers is the crew.
Elastic sees the symptom in your logs and your search traffic. Data Workers fixes the data behind it and proves the fix. Here is a Thursday on a commerce search team. This is an illustrative example.
| Time | System | What happens |
|---|---|---|
| Thu 01:10 | Akeneo + Prefect | The product team renames the attribute price_eur to price in Akeneo; the nightly Prefect sync lands the new column in Databricks |
| 02:00 | Databricks + dbt | dbt build fails the not_null test on stg_products.price_eur; dbt skips the downstream model product_docs. The flow treats test failures as warnings, so the Prefect run ends Completed |
| 02:40 | Elasticsearch | The reindex_product_docs flow reads product_docs, finds no new rows and indexes nothing. The index products-rag still holds Wednesday's catalog, without 1,412 new products or the prices for tomorrow's sale |
| 09:20 | Kibana | An Elasticsearch query rule on the logs-shop-assistant-* data stream fires: fallback answers are up six times; a case opens for the search on-call |
| 09:31 | Agent Builder | The on-call asks Agent Builder; a query shows the newest document in products-rag is 31 hours old and the reindex flow's logs say "0 documents updated" |
| 09:44 | Data Workers | Takes the failed not_null test from the analytics engineer and the Akeneo rename the product team confirms, traces lineage from raw.akeneo_products through stg_products and product_docs to the reindex flow, and maps the blast radius: the index, the assistant and the merchandising dashboard |
| 09:52 | Data Workers | Proposes the fix: a diff to stg_products that maps price and keeps the test, for the model owner to merge, plus a run plan: rebuild the two models, then rerun the reindex flow |
| 10:15 | Spellbook | The product data owner reviews the diff and the blast radius and approves; the analytics engineer merges the diff |
| 10:18 | Prefect | Data Workers queues the dbt job and then the reindex flow through Prefect |
| 11:06 | Databricks + dbt | The rebuild finishes; the analytics engineer reports the not_null test passing and product_docs holding the new products |
| 11:12 | Elasticsearch | The on-call's client asks the Elastic MCP server for the index's document count and newest timestamp; both match product_docs |
| 11:20 | Kibana | The on-call adds the link to the Data Workers receipt in Spellbook as a case comment and closes the case |
Elastic did its job: the rule caught the bad answers, the case found the right person, and Agent Builder found the stale index by 09:31. The cause lived three systems upstream, and Data Workers had the index current and verified before the sale weekend.
| Job | What Elastic does | What Data Workers does |
|---|---|---|
| Detection | Kibana rules and Streams significant events watch logs, traces and metrics for errors and anomalies | Watches freshness and quality on the tables that feed the index and reads schema changes from the dbt manifest and pull requests, so the break is caught upstream |
| Coordination | Alerts open cases, connectors notify Slack, PagerDuty or email, and the case tracks the work | Routes each data fix to the named owner of the model or table, with the blast radius attached |
| Investigation | Agent Builder queries indices, reads logs, traces and alerts, and finds what changed inside Elastic | Traces the break into the data: which column, which model, which flow, which consumers, which owner |
| The fix | Agent Builder can run workflows and update cases, with tool confirmation where the tool asks for it | Proposes the data change (model diff, rerun, backfill with a recorded undo step), gets a named approval and queues the run through the orchestrator |
| Verification | The rule stops firing and the case closes | Confirms the numbers: the owner reports the tests passing on the source table; the index count is confirmed through the Elastic MCP server in the same client |
| The record | The case, its comments and attachments, and the logs | A receipt in Spellbook for the data change: cause, diff, approver, what it touched, how it was verified, how to undo it |
Why doesn't Elastic just do this itself?
Because Elastic built its agent layer around the data it holds, and that is the right focus for a platform that stores every log, trace and security event you have. Agent Builder agents "answer questions and take actions over your Elasticsearch data." Its built-in tools that change things act inside Elastic: platform.core.cases.manage creates and updates cases, platform.streams.update_stream changes a stream's processing and "requires user confirmation before applying changes", and workflows run steps you defined. Elastic is careful about what confirmation means: human-in-the-loop prompts "do not replace role-based access control or grant additional privileges." Attack Discovery follows the same line in security: it groups alerts into attack narratives so analysts can prioritize, and the response stays with the analyst.
A data fix is a different job with a different liability. The renamed attribute needs a mapping change in a dbt model the analytics engineer owns. The rebuild runs in Databricks under the data team's permissions, through the Prefect deployment the data team operates. Lineage has to show every consumer of product_docs, including the dashboard nobody on the search team knows exists. None of those systems belong to Elastic. Data Workers is built for that job.
Every tool owns a slice. Data Workers covers the whole lifecycle
Each point tool adds another console, another contract and another handoff. Data Workers covers the whole lifecycle with one context, one approval flow and one audit trail, and builds on Elastic where your engineers already look.
| Stage | Data Workers | Elastic | Why we scored it this way |
|---|---|---|---|
| Catalog & Context | 9 | 3 | Elasticsearch indexes and retrieves documents, and Agent Builder explores indices and mappings. Data Workers keeps one governed context graph of definitions, owners, lineage, quality and usage across every platform. |
| Analytics & Insights | 8 | 5 | Kibana dashboards, Discover and Elastic's piped query language answer questions over indexed data, mostly logs, metrics and events. Data Workers answers business data questions from governed definitions with lineage behind every number. |
| Data Quality | 8 | 3 | Streams reports failed and degraded documents for log ingestion. Data Workers writes, runs and repairs the checks on warehouse tables and dbt models. |
| Observability & Incidents | 8.5 | 9 | Elastic's home stage: Kibana rules, alerts and cases, APM, Streams significant events, and Agent Builder investigating across logs, traces and alerts. Data Workers diagnoses the data incident across lineage, fixes it with approval and verifies it. |
| Pipelines & Ingestion | 8.5 | 3 | Ingest pipelines and Streams processing parse logs on the way in. Data Workers repairs the data pipeline itself: dbt models, reruns and backfills, with approvals. |
| Schema & Migration | 8 | 1 | Index mappings and Streams schemas cover documents in Elasticsearch, not warehouse tables. Data Workers catches upstream schema changes, scores their blast radius and plans migrations in waves. |
| Governance & Access | 8.5 | 3 | Spaces, roles and API keys govern who sees which indices. Data Workers dry-runs each warehouse access request and routes it to a named approver, recording the grant and its expiry. |
| Security & Privacy | 8 | 8.5 | Elastic's second home stage: Elastic Security is a SIEM with detection rules and Attack Discovery. Data Workers' pull request review flags new columns whose names or annotations look sensitive, and leaves a receipt on every data change. |
| Cost / FinOps | 8 | 2 | Index lifecycle policies manage storage tiers for Elastic's own data. Data Workers attributes Snowflake spend to the dbt model behind it and reads AWS Cost Explorer by service. |
| MLOps & Models | 7.5 | 3 | Machine learning jobs find anomalies in telemetry, and inference endpoints serve embedding models. Data Workers keeps the tables under your models and your RAG index fresh and correct. |
Elastic leads where it should, in observability and security. For watching systems versus fixing data, read Data Workers vs data observability and what is an agentic data platform.
Where Elasticsearch as a vector store fits. semantic_text, hybrid search and Jina models on Elastic Inference Service stay Elastic's job. A RAG index is only as good as the warehouse tables it is built from, and Data Workers keeps those fresh and correct. The same split applies on dedicated vector databases: see you're on Pinecone and vector databases for data engineers.
How Elastic and Data Workers work together
Elastic stays on top, where rules fire, cases open and Agent Builder investigates. Spellbook Data Catalog (in preview) is where the data team reviews, approves and rolls back changes. Between them, Data Context Wizard keeps one governed context graph, the Data-Agents Swarm does the work with 20 specialist agents, and the Autonomous Data-Conductor runs each fix end to end: detect, diagnose, fix, review, verify, remember.
How they connect today. Data Workers connects to Elastic over the Elastic MCP server or the Elasticsearch API. The Elastic MCP server is the Agent Builder endpoint at {KIBANA_URL}/api/agent_builder/mcp, generally available on Elastic Stack since 9.3 and on Serverless; Elastic deprecated its standalone Elasticsearch MCP server in favour of it. A client can do what the exposed tools allow: an API key is one shared identity; on Serverless, OAuth 2.1 lets each person act with their own permissions.
The data work. diagnose_incident and get_root_cause tie the failed test to the renamed attribute, and trace_cross_platform_lineage and blast_radius_analysis map what sits downstream. explain_table returns the definition, lineage and documentation of product_docs, and get_incident_history checks for repeats. Verification uses the monitor_metrics baseline and the owner's test, and every step lands in get_audit_trail. Databricks, dbt and Prefect are native connectors. Akeneo connects over its API today. export_otel_spans emits the agents' activity as OpenTelemetry spans, so your SRE team can watch Data Workers in Kibana.
Side by side in one client. Every Data Workers agent is an MCP server; the client setup guide documents the path: clone the open-source repo and add one start-agent.sh entry per agent.
Example: a .cursor/mcp.json; the Elastic entry follows Elastic's documented mcp-remote setup with an API key.
{
"mcpServers": {
"elastic-agent-builder": {
"command": "npx",
"args": ["mcp-remote", "${KIBANA_URL}/api/agent_builder/mcp", "--header", "Authorization:${AUTH_HEADER}"],
"env": { "KIBANA_URL": "${KIBANA_URL}", "AUTH_HEADER": "ApiKey ${API_KEY}" }
},
"dw-catalog": { "command": "/path/to/dataworkers-claw-community/start-agent.sh", "args": ["dw-context-catalog"] },
"dw-schema": { "command": "/path/to/dataworkers-claw-community/start-agent.sh", "args": ["dw-schema"] },
"dw-incidents": { "command": "/path/to/dataworkers-claw-community/start-agent.sh", "args": ["dw-incidents"] },
"dw-quality": { "command": "/path/to/dataworkers-claw-community/start-agent.sh", "args": ["dw-quality"] }
}
}Ask "why is the shop assistant missing new products?" and the client calls both: Elastic returns the alert and the index's newest timestamp; Data Workers returns the renamed attribute, the skipped model, the blast radius and a proposed fix. Data Workers' guardrail decides whether a data change may run, with a named approver. If you expose Agent Builder's case tools, the same client can add the receipt link to the case. Give the Elastic API key only the indices and tools the client needs.
From inside Kibana. Agent Builder can also import tools from external MCP servers through an MCP connector (Preview since 9.3), so a team can add Data Workers' read tools to the agent it already uses. The Data Workers remote endpoint serves /mcp with an API key (bearer) or OAuth tokens from your identity provider, verified through JWKS. Data changes still route to Spellbook for a named approval.
One stale index, L0 to L4. The same incident on products-rag, at each level of the autonomy ladder, set per domain.
- •L0 manual. The search engineer hunts through Prefect, dbt and Databricks by hand.
- •L1 observe. Data Workers writes the diagnosis and blast radius to Spellbook for the case to link. No data changes.
- •L2 propose. Data Workers proposes the
stg_productsdiff and the run plan. Nothing runs until the product data owner approves in Spellbook. - •L3 act reversibly. For change classes with a proven record, such as rerunning a model and its reindex flow after an approved fix, Data Workers queues the runs, verifies them and records the receipt.
- •L4 autonomous. In a scoped domain such as the product catalog, every skipped model that feeds an index triggers a diagnosis, and approved fix classes run end to end.
For the safety model, read is it safe to let AI agents change production data, how approvals work and autonomy levels L0 to L4. On where data lives: the agents run in your infrastructure, your data stays in your systems, and the hosted Conductor sees workflow metadata only.
What changes for your team
- •Incidents. A Kibana alert that traces back to data gets a diagnosis, a fix and a receipt before the case closes.
- •Data quality. Each incident becomes a check on the model that broke, so the next stale index is caught upstream.
- •Cloud spend. Warehouse spend is traced to the dbt model behind it, and warehouse settings are drafted for the owner to apply.
- •Access. Warehouse access becomes a scoped, time-boxed grant the data owner approves.
- •Audits. Elastic keeps the logs and the case; Data Workers records what changed in the data and who approved it.
- •Migrations. A warehouse or pipeline move is planned in waves with parity checks, and the owner signs off each one.
See data freshness monitoring, agentic RAG for data engineering, the data incident response playbook and who owns the agents.
Keep Elastic, or consolidate?
Keep Elastic if you love it; Data Workers works with it from day one. Many teams consolidate once Data Workers runs that slice too.
For almost every Elastic customer the answer is to keep it. What teams consolidate is the stack around the warehouse: a separate data observability contract, a data-quality tool, a stale catalog and the scripts that check whether an index caught up. Weighing a build on the Agent Builder MCP server and a coding agent? Read build it ourselves with Claude Code and MCP servers: the MCP calls are the easy part; the context graph, approvals and rollback are the work. Across the incident stack, see you're on New Relic, you're on Datadog, you're on PagerDuty and Data Workers integrations.
The case for your CFO
The outcome: Elastic sees problems in your systems and powers search. Data Workers makes sure the data behind that search is right, so a renamed attribute does not become a weekend of wrong prices in the shop assistant.
The risk story is plain. Autonomy is set per domain: at L1 Data Workers changes no data; at L2 it changes nothing until a named person approves; at L3 it applies changes it can undo. An unanswered approval request expires and escalates, never auto-grants. No agent can promote its own work. Every change carries a receipt: cause, diff, approver, what it touched, how it was verified and how to undo it. An org-wide stop halts all autonomous dispatch. Zero migration: Elastic, Akeneo, Prefect, Databricks and dbt stay where they are.
Why now: Agent Builder finds what changed inside Elastic quickly, and RAG has put warehouse tables behind customer-facing answers, so the slow part is now upstream. The first win is read-only: every Kibana alert that traces to a data source gets a diagnosis and a blast radius, and every table feeding an index gets a freshness check. Your rules, cases, Spaces, warehouse permissions and code review stay the same. For the numbers, see the ROI of agentic data operations.
The sentence to repeat upstairs: "Elastic tells us something looks wrong; Data Workers fixes the data behind it and proves it, with an approval and a receipt for every change."
Getting started
Start with a pilot. Pick one domain whose symptoms show up in Kibana, such as the product catalog behind your RAG index, connect Data Workers to your warehouse, dbt project and orchestrator, add the Elastic MCP server beside it, and run at L1 so every relevant alert gets a diagnosis and a blast radius. Then turn on the first write class at L2, such as model reruns and reindex runs after an owner approves the fix. The pilot path and plans are on the pricing page, and the pilot is credited in full against the first year.
FAQ
Does Data Workers have an Elastic connector? Data Workers connects to Elastic over the Agent Builder MCP server or the Elasticsearch API today, running beside Elastic in the same client. It also emits its agent activity as OpenTelemetry spans that Elastic Observability ingests.
How is this different from Elastic Agent Builder? Agent Builder is Elastic's agent layer over your Elasticsearch data: it queries, searches and acts inside Elastic, such as updating a case or running a workflow. Data Workers works on the data upstream of Elastic: warehouse tables, dbt models and orchestrator runs, each change with approval, rollback and a receipt.
Elastic deprecated the AI Assistant and its standalone MCP server. Does that change anything? No. The Observability and Search AI Assistant is deprecated as of Elastic 9.4, and Agent Builder is the default chat experience there and in Elastic Security. The standalone Elasticsearch MCP server receives only critical security updates; Elastic points to the Agent Builder endpoint at {KIBANA_URL}/api/agent_builder/mcp, which is the one to use beside Data Workers.
We use Elasticsearch as our vector store for RAG. Where does Data Workers fit? Upstream of the index. Elastic handles embeddings, chunking, hybrid retrieval and reranking. Data Workers keeps the source tables fresh and correct, catches schema changes before they reach the reindex job, and verifies the source after a fix so the index catches up.
Does Data Workers change anything in our Elastic deployment? No. Your rules, cases, data streams, indices and Agent Builder agents stay as they are, and the search team keeps running reindex jobs through its own flows. Data Workers works on the data systems behind Elastic.
Sources
- •Elastic Docs, Elastic Agent Builder (GA on Elastic Stack since 9.3, Preview in 9.2; GA on Serverless), https://www.elastic.co/docs/explore-analyze/ai-features/elastic-agent-builder (checked Oct 2, 2026)
- •Elastic Docs, Elastic Agent Builder MCP server (endpoint, API key and OAuth 2.1 on Serverless, full tool catalog), https://www.elastic.co/docs/explore-analyze/ai-features/agent-builder/mcp-server (checked Oct 2, 2026)
- •Elastic Docs, Authenticate MCP clients with API keys (mcp-remote configuration), https://www.elastic.co/docs/explore-analyze/ai-features/agent-builder/mcp-server-api-keys (checked Oct 2, 2026)
- •Elastic Docs, MCP tools in Elastic Agent Builder (importing tools from external MCP servers, Preview since 9.3), https://www.elastic.co/docs/explore-analyze/ai-features/agent-builder/tools/mcp-tools (checked Oct 2, 2026)
- •Elastic Docs, Agent Builder built-in tools reference (cases.manage and streams.update_stream, Preview since 9.5), https://www.elastic.co/docs/explore-analyze/ai-features/agent-builder/tools/builtin-tools-reference (checked Oct 2, 2026)
- •Elastic Docs, Chat with Agent Builder agents (human-in-the-loop prompts, GA since 9.4), https://www.elastic.co/docs/explore-analyze/ai-features/agent-builder/chat (checked Oct 2, 2026)
- •Elastic Docs, Elastic AI Assistant for Observability and Search (deprecated since 9.4), https://www.elastic.co/docs/solutions/observability/ai/observability-ai-assistant (checked Oct 2, 2026)
- •Elastic Docs, Elastic AI Assistant for Security (Agent Builder default chat from 9.4), https://www.elastic.co/docs/solutions/security/ai/ai-assistant (checked Oct 2, 2026)
- •Elastic Docs, Attack Discovery, https://www.elastic.co/docs/solutions/security/ai/attack-discovery (checked Oct 2, 2026)
- •Elastic Docs, Streams (GA since 9.2; significant events, data quality), https://www.elastic.co/docs/solutions/observability/streams/streams (checked Oct 2, 2026)
- •Elastic Docs, Workflows (GA since 9.4, Preview in 9.3), https://www.elastic.co/docs/explore-analyze/workflows (checked Oct 2, 2026)
- •Elastic, Kibana release notes (9.5.0: cases skill and tools in Agent Builder, tracing on by default), https://www.elastic.co/docs/release-notes/kibana (checked Oct 2, 2026)
- •Elastic, Elasticsearch vector database (semantic_text, hybrid search, Jina models via Elastic Inference Service), https://www.elastic.co/elasticsearch/vector-database (checked Oct 2, 2026)
- •Elastic, Elasticsearch MCP server repository (deprecated, superseded by the Agent Builder MCP endpoint), https://github.com/elastic/mcp-server-elasticsearch (checked Oct 2, 2026)
- •Data Workers open-source repository, https://github.com/DataWorkersProject/dataworkers-claw-community (checked Oct 2, 2026)
- •Data Workers client setup guide, https://dataworkers.io/opensource-docs/client-setup/ (checked Oct 2, 2026)