Security and observability data teams: stop paying to store noise, start trusting your telemetry

Key takeaways

• •Who this is for. Teams that own logs, metrics, traces and security events and the pipelines into SIEM, observability tools and a security data lake.
• •The problem today. Detections that go blind after vendor updates, secrets found in the SIEM and ingest bills that grow every month.
• •What changes. Every log source is checked for format changes, so a renamed field never blinds a detection for days. Sensitive data is classified as logs land, so secrets never reach the SIEM.
• •What stays with your people. Changes to detection inputs and redaction rules: these are security controls, so detection engineering approves.
• •The result. Security teams that trust their telemetry and stop paying to store noise.

The PRD for agentic security and observability data teams

This is the requirements document we would write with you for your team. Everything below expands on it.

What going fully agentic and autonomous means for security and observability data teams

• •Put your data pipeline on autopilot. Every log source is checked for format changes, so a renamed field never blinds a detection for days.
• •Run the data back office without the busywork. The work that fills the week today moves to agents: detections that go blind after vendor updates, secrets found in the SIEM and ingest bills that grow every month.
• •Scale data without scaling headcount. Sensitive data is classified as logs land and volume anomalies are routed to cheaper storage, so the data you manage grows without the team having to.
• •Get the plumbing off your plate. Security teams that trust their telemetry and stop paying to store noise.

Who this is for

Teams that own logs, metrics, traces and security events and the pipelines into SIEM, observability tools and a security data lake. They sit in security engineering or platform/SRE. Titles include Security Data Engineer, Detection Engineering Lead and Observability Platform Owner.

The shift

Telemetry volume grows faster than budgets, and security teams increasingly move it into lakes they can query cheaply. Every new source needs parsing, enrichment and redaction, and every schema change can blind a detection. Agents can keep telemetry pipelines parsed, redacted and right-sized.

The product you ship

• •Parsed, normalized security and ops events
• •Detection-ready datasets
• •A security data lake with retention tiers
• •Cost-controlled routing to SIEM and observability tools

Where the week goes

• •A log source changes format and a detection goes blind.
• •Sensitive data appears in logs.
• •Ingest costs spike.
• •New sources need parsers.
• •Retention rules drift.

Add it up and upkeep is most of the team's week. Here is how a typical week splits today, and once agents own the back office:

Time spent on new data products moves from 14% of the week to 50%. Most of the rest becomes review: people steer and approve agent work instead of doing it by hand.

Then and now

These are the impact metrics we move for security and observability data teams:

• •Cost spike routed: from about 2 weeks to under 1 h
• •Format change mapped: from about 3 days to 15-30 min
• •Retention enforced: from about a month to hours

Across the routine work in this domain:

The biggest change is Cost spike routed: from about 2 weeks to under 1 h.

Use cases, in depth

A firewall log format change

Today. A vendor update renames fields; a detection rule stops matching.

With Data Workers. Schema Evolution detects the change, Pipeline Building updates the parser, Quality Monitoring confirms detection inputs recover.

What changes. Detections never go blind.

Secrets in application logs

Today. A release starts logging tokens.

With Data Workers. scan_pii classifies, review_pii_writes flags the release, redaction proposed.

What changes. Secrets never reach the SIEM.

A cost spike from debug logs

Today. A service ships debug logging to production.

With Data Workers. Usage Intelligence detects the volume anomaly; Cost Savings proposes routing to cold storage.

What changes. The bill stays flat.

Agents working together: A firewall update that blinded a detection

Minute by minute

Illustrative example. Firewall logs flow through a telemetry pipeline into a security lake and SIEM.

Without the agents, the detection is blind until someone notices.

The agents on the job

Data Context Wizard keeps one governed graph of definitions, lineage and owners that every agent and person answers from. Autonomous Data-Conductor runs each job from detection to a verified result. Spellbook Data Catalog is where your team approves, audits and rolls back, and asks questions in plain English.

What goes on autopilot

Once a domain has earned it, this work runs on its own, and each item removes a problem your team deals with today:

• •Every log source is checked for format changes, so a renamed field never blinds a detection for days.
• •Sensitive data is classified as logs land, so secrets never reach the SIEM.
• •Volume anomalies are routed to cheaper storage, so ingest cost stops climbing.

Where your people stay in charge

Agents never get blanket access. Each domain has its own autonomy level, set by you, and it only rises when the receipts show the agents are reliable. Everything starts at L1, observing only; reversible upkeep such as reruns, staging fixes, documentation and cleanup is usually first to run on its own.

• •Changes to detection inputs and redaction rules always goes to a named approver: these are security controls, so detection engineering approves.
• •Incident response decisions: these stay with people.

Once a deployment has matured, here is how that ladder plays out across task classes:

Retention upkeep closes autonomously 88% of the time. For Detection inputs, autonomous closure is 0% by design.

What happens when an agent is wrong

• •At L2 propose, nothing changes until a named person approves. The proposal shows the full diff, the blast radius and how to roll it back. Unanswered requests expire and escalate.
• •At L3 act, reversibly, every change keeps a rollback point, and Conductor verifies the downstream number after the change. If verification fails, the change is reversed and the domain drops a level.
• •No agent approves or promotes its own work. Autonomy only grows on evidence, per repair class, and repeated errors shrink scope automatically.
• •One org-wide stop halts all autonomous dispatch, and every action lands in a hash-chained audit trail you can export with generate_audit_report.

How it fits your stack

Data Workers connects to what you already run: Kafka; Cribl or Vector; Splunk, Sentinel or Chronicle; a security lake on S3/Iceberg or Snowflake; Datadog or Grafana. Each agent is an MCP server, so the same agents work inside Claude Code, Cursor or Codex and in Slack or Teams through Spellbook. Nothing migrates.

• •Runs in your infrastructure with your credentials and your model key. The hosted Autonomous Data-Conductor sees workflow metadata only, never rows.
• •Uses the access you already govern. Agents act under scoped service identities and your existing roles; every grant goes through provision_access and is logged.
• •Proposes changes as diffs in your dbt project, orchestrator and warehouse, with blast radius, tests and a rollback point, and applies them only at the autonomy level you set for that domain.
• •Open-source core. The Apache-2.0 community edition ships 11 agents and 160+ MCP tools, so your engineers can read exactly what the agents call.

Governance and regulation

Security teams must protect sensitive data in logs and keep audit evidence. Data Workers classifies and redacts, keeps lineage and logs every change.

Data Workers doesn't make you compliant; your controls, people and assessors do. It supplies controls and evidence wherever agents touch data. Not legal advice.

The alternatives security and observability data teams weigh

Build it yourselves. A strong team can wire a coding agent to MCP servers for the warehouse and dbt. The work is everything around it: routing approvals to the right owner, blast radius, rollback, receipts, and current context about every other system.

Observability, catalogs and point tools. Observability alerts, catalogs document, and quality tools test. Data Workers takes their signals, diagnoses, proposes the fix, verifies it and leaves the receipt.

Platform-native agents. Snowflake made its managed MCP server and Cortex Agents generally available on Nov 4, 2025, and Databricks made Genie One generally available on Jun 16, 2026. They are excellent inside their platform; this work crosses the source systems, orchestrator, warehouse and BI, and Data Workers works across all of them.

In this space specifically. Telemetry pipelines route data; Data Workers also detects format changes, fixes parsers and verifies detection inputs.

The case for your CTO

What this replaces. Detections that go blind after vendor updates, secrets found in the SIEM and ingest bills that grow every month.

What your team gets back. Security teams that trust their telemetry and stop paying to store noise.

Integration effort. Connecting is configuration, not a migration. The agents attach over MCP to the warehouse, orchestrator, dbt project and source systems you already run, under the roles you already govern, inside your own infrastructure. Your engineers can read every tool the agents call in the open-source core.

Risk and control. Autonomy is set per domain. Everything starts at L1, observing only; reversible work moves up only when the receipts justify it; changes to detection inputs and redaction rules stay at L2 with a named approver; and one org-wide stop halts all autonomous work.

Why now. Telemetry grows faster than security budgets.

The first win. Format-change detection and cost routing on the top 20 sources. The pilot below lays out the scope, what you see each week and the cost.

The climb: from assisted to autonomous to agentic

Our thesis is that the data platform climbs the way analytics did, in stages, and that the hard part of data work was never knowing what to do but doing it. For security and observability data teams, the climb looks like this:

• •Assisted. Agents work alongside your team. They read the whole stack, show what needs doing and propose every fix with a receipt; changes to detection inputs and redaction rules always go to a named approver.
• •Autonomous, under your governance. Routine work runs on its own: every log source is checked for format changes, sensitive data is classified as logs land, and volume anomalies are routed to cheaper storage. Your team governs the agents instead of doing the upkeep.
• •The agentic enterprise. Security data teams will run telemetry like a governed product: every source parsed, redacted and routed by value, and detections protected from upstream change. Cost per detection will fall even as sources multiply.

You choose the altitude, every step is visible and reversible, and the pilot below is the first rung.

The pilot: how security and observability data teams start

The pilot is how your team proves the value on its own data before committing.

• •What we connect. Kafka; Cribl or Vector; Splunk, Sentinel or Chronicle; a security lake on S3/Iceberg or Snowflake; Datadog or Grafana, over MCP, inside your infrastructure. Nothing migrates.
• •Scope. Format-change detection and cost routing on the top 20 sources.
• •First week. Agents run at L1 across the scoped domain. They map every pipeline, definition and lineage path and show the fixes they would have made, with receipts, so your team sees the value before anything changes.
• •By week two. Every log source is checked for format changes and sensitive data is classified as logs land start running on reversible work. Changes to detection inputs and redaction rules stay at L2 with a named approver.
• •At day 30. We review the impact metrics against your baseline:

• •Cost spike routed: from about 2 weeks to under 1 h
• •Format change mapped: from about 3 days to 15-30 min
• •Retention enforced: from about a month to hours
• •the share of retention upkeep closed autonomously, targeting 88% as the deployment matures

• •What you decide. Whether to expand to the next domain. The pilot fee is credited in full against the first year.
• •Cost. Start with a pilot ($7,500 one-time, credited in full against the first year). Scale is from $1,000 a month and Enterprise from $3,000 a month, billed annually, with unlimited seats, no usage meter and no markup on model spend. See pricing.

The pilot is the start of a longer curve:

By month 12, more than 44% of agent actions run fully autonomously while detection inputs stay with a named approver.

FAQ

Does it replace Cribl or our SIEM? No; it keeps what flows through them correct.

Can it redact automatically? Redaction rules are proposed at L2.

Does it watch costs? Yes, by volume per source.

Where do the agents run? In your infrastructure, with your credentials and model key. The hosted Conductor sees workflow metadata only.

Do we have to migrate anything? No. Your warehouse, dbt project, orchestrator, catalog and BI stay where they are.

Sources

• •CB Insights, Cribl company profile: https://www.cbinsights.com/company/cribl
• •Data Workers pricing: https://dataworkers.io/pricing/
• •Data Workers open-source repository (tool registrations): https://github.com/DataWorkersProject/dataworkers-claw-community
• •Snowflake, managed MCP server and Cortex Agents GA release note (Nov 4, 2025): https://docs.snowflake.com/en/release-notes/2025/other/2025-11-04-cortex-agents-mcp
• •Databricks, Genie One launch (Jun 16, 2026): https://www.databricks.com/company/newsroom/press-releases/databricks-launches-genie-one-all-new-agentic-coworker-every-team

Related: Best practices for agentic security and observability telemetry