Best practices for agentic security and observability telemetry: a field guide for security and observability data teams
How security and observability data teams move their data back office to agents, practice by practice: the state of the space, the old world and the new, what the agents do step by step, how to set autonomy, who owns what, and how to start.
Telemetry is the most voluminous data most companies have.
Parsing, redaction and routing are repeatable; agents should own them.
The state of the space
Teams that own logs, metrics, traces and security events and the pipelines into SIEM, observability tools and a security data lake. They sit in security engineering or platform/SRE. Titles include Security Data Engineer, Detection Engineering Lead and Observability Platform Owner.
Telemetry volume grows faster than budgets, and security teams increasingly move it into lakes they can query cheaply. Every new source needs parsing, enrichment and redaction, and every schema change can blind a detection. Agents can keep telemetry pipelines parsed, redacted and right-sized.
The old world
A firewall update renames fields; a detection stays blind for three days.
It is rarely one failure. Three patterns repeat:
- •A firewall log format change. A vendor update renames fields; a detection rule stops matching.
- •Secrets in application logs. A release starts logging tokens.
- •A cost spike from debug logs. A service ships debug logging to production.
The new world
The rename was caught at 02:05 and the parser approved by 06:30.
Here is that new world as a chain of hand-offs between agents:
These are the impact metrics we move:
- •Cost spike routed: from about 2 weeks to under 1 h
- •Format change mapped: from about 3 days to 15-30 min
- •Retention enforced: from about a month to hours
The rest of this report walks through the 5 practices that make that possible. At a glance:
The practices, one by one
1. Detect format changes per source
Blind detections are the real risk.
Why it is harder than it looks. Vendors change log formats in routine updates without notice.
The old world. Found days later.
The new world. detect_schema_change on every stream.
What the agents do, step by step:
- 1.The Schema Evolution agent calls
detect_schema_changeand catches format changes. - 2.The Pipeline Building agent calls
validate_pipelineand updates parsers. - 3.The Quality Monitoring agent calls
run_quality_checkand checks detection inputs.
In practice. Caught in minutes.
What to measure. Hours a detection is blind.
2. Redact on arrival
Secrets in logs spread fast.
Why it is harder than it looks. Secrets and personal data appear wherever developers log.
The old world. Reviews.
The new world. scan_pii and redaction.
What the agents do, step by step:
- 1.The Data Security agent calls
scan_piiand classifies fields. - 2.The Data Security agent calls
review_pii_writesand flags releases. - 3.The Data Access & Governance agent calls
request_governance_reviewand proposes redaction.
In practice. Tokens never reach the SIEM.
What to measure. Sensitive findings in SIEM.
Practices like these compound. Across the routine work they cover:
3. Route by value
Not every log needs hot storage.
Why it is harder than it looks. Not every log needs hot, expensive storage.
The old world. Everything hot.
The new world. Cost Savings routing.
What the agents do, step by step:
- 1.The Usage Intelligence agent calls
detect_usage_anomaliesand finds volume spikes. - 2.The Cost Savings & Data Cleanup agent calls
get_cost_enrichmentand prices each source. - 3.The Cost Savings & Data Cleanup agent calls
detect_dead_assetsand finds unused data.
In practice. Debug logs to cold storage.
What to measure. Ingest cost per GB.
4. Onboard sources as missions
Parsers are repetitive.
Why it is harder than it looks. Parsers are repetitive and always behind.
The old world. Weeks.
The new world. Autopilot.
What the agents do, step by step:
- 1.The Pipeline Building agent calls
generate_pipelineand builds parsers. - 2.The Quality Monitoring agent calls
create_quality_tests_for_pipelineand tests them. - 3.The Data Context & Catalog agent calls
generate_documentationand documents sources.
In practice. 30 sources in a week.
What to measure. Days per source.
Autonomy where work is reversible, a named approver where it matters:
5. Hold detection inputs for a person
Detections are security controls.
Why it is harder than it looks. Detection inputs are security controls.
The old world. Mixed.
The new world. L2.
What the agents do, step by step:
- 1.The Data Context & Catalog agent calls
blast_radius_analysisand shows affected detections. - 2.The Autonomous Data-Conductor calls
request_governance_reviewand holds at L2. - 3.The Data Access & Governance agent calls
generate_audit_reportand records it.
In practice. Every change approved.
What to measure. Changes with approver.
Designing the autonomy ladder
Autonomy is set per task class, not per tool. This is where we would set each class for a mature deployment, and why:
| Task class | Closed autonomously | Escalated | Handled by a person | Where we set it | Why |
|---|---|---|---|---|---|
| Retention upkeep | 88% | 8% | 4% | L4 once the receipts are clean | The work is reversible and repeatable |
| Parser updates | 68% | 22% | 10% | L3: acts and keeps a rollback point | Most cases are routine; the rest escalate with evidence |
| Volume checks | 82% | 12% | 6% | L4 once the receipts are clean | The work is reversible and repeatable |
| Redaction rules | 30% | 56% | 14% | L3 for routine cases, L2 for the rest | Judgment is needed often enough to keep a person close |
| Detection inputs | 0% | 86% | 14% | L2: always proposed to a named approver | The number carries accountability outside the data team |
The operating model: who owns what
Agents change who does the work, not who is accountable for it:
| Role | Owns | What the agents take off their plate |
|---|---|---|
| Head of Security Data | Standards, autonomy levels per domain and which domain goes next | Assembling status, chasing owners, keeping the catalog current |
| Domain owners and approvers | Changes to detection inputs and redaction rules | Preparing the diff, blast radius and rollback plan for every proposal |
| Data engineers | Reviewing agent proposals and building new data products | Every log source is checked for format changes; sensitive data is classified as logs land |
The swarm behind the practices
| Agent | What it does for you | Tools it calls |
|---|---|---|
| Schema Evolution | Detects log format changes | detect_schema_change, validate_schema_compatibility, generate_migration, rollback_migration |
| Pipeline Building | Parsers and normalization with tests | generate_pipeline, validate_pipeline, sandbox_test, deploy_pipeline |
| Data Security | Classifies and redacts sensitive data | scan_pii, review_pii_writes, pci_card_data_masking |
| Quality Monitoring | Volume and detection-input checks | set_sla, run_quality_check, get_anomalies, create_quality_tests_for_pipeline |
| Usage Intelligence | Volume anomalies by source | detect_usage_anomalies, get_adoption_dashboard, get_cost_enrichment |
| Cost Savings & Data Cleanup | Routing and retention tiers | get_snowflake_usage, get_cost_enrichment, detect_dead_assets |
Data Context Wizard keeps one governed graph of definitions, lineage and owners that every agent and person answers from. Autonomous Data-Conductor runs each job from detection to a verified result. Spellbook Data Catalog is where your team approves, audits and rolls back, and asks questions in plain English.
What disappears
- •Blind detections after vendor updates.
- •Secrets in the SIEM.
- •Surprise ingest bills.
- •Parser backlogs.
- •Retention drift.
The magic is not any single agent. It is that the back office keeps running: bad data is caught where it enters, every new table gets quality rules and documentation without anyone writing them, access follows policy, fixes are verified before anyone opens a dashboard, and every action leaves a signed receipt.
A product manager's scorecard
| Metric | How to read it |
|---|---|
| Hours a detection is blind | Baseline it in the first week of the pilot; review it monthly. |
| Sensitive findings in SIEM | Baseline it in the first week of the pilot; review it monthly. |
| Ingest cost per GB | Baseline it in the first week of the pilot; review it monthly. |
| Days per new source | Baseline it in the first week of the pilot; review it monthly. |
| Changes with approver | Baseline it in the first week of the pilot; review it monthly. |
Where this goes next
Security data teams will run telemetry like a governed product: every source parsed, redacted and routed by value, and detections protected from upstream change. Cost per detection will fall even as sources multiply.
The climb: from assisted to autonomous to agentic
Our thesis is that the data platform climbs the way analytics did, in stages, and that the hard part of data work was never knowing what to do but doing it. For security and observability data teams, the climb looks like this:
- •Assisted. Agents work alongside your team. They read the whole stack, show what needs doing and propose every fix with a receipt; changes to detection inputs and redaction rules always go to a named approver.
- •Autonomous, under your governance. Routine work runs on its own: every log source is checked for format changes, sensitive data is classified as logs land, and volume anomalies are routed to cheaper storage. Your team governs the agents instead of doing the upkeep.
- •The agentic enterprise. Security data teams will run telemetry like a governed product: every source parsed, redacted and routed by value, and detections protected from upstream change. Cost per detection will fall even as sources multiply.
You choose the altitude, every step is visible and reversible, and the pilot below is the first rung.
The pilot: how security and observability data teams start
The pilot is how your team proves the value on its own data before committing.
- •What we connect. Kafka; Cribl or Vector; Splunk, Sentinel or Chronicle; a security lake on S3/Iceberg or Snowflake; Datadog or Grafana, over MCP, inside your infrastructure. Nothing migrates.
- •Scope. Format-change detection and cost routing on the top 20 sources.
- •First week. Agents run at L1 across the scoped domain. They map every pipeline, definition and lineage path and show the fixes they would have made, with receipts, so your team sees the value before anything changes.
- •By week two. Every log source is checked for format changes and sensitive data is classified as logs land start running on reversible work. Changes to detection inputs and redaction rules stay at L2 with a named approver.
- •At day 30. We review the impact metrics against your baseline:
- •Cost spike routed: from about 2 weeks to under 1 h
- •Format change mapped: from about 3 days to 15-30 min
- •Retention enforced: from about a month to hours
- •the share of retention upkeep closed autonomously, targeting 88% as the deployment matures
- •What you decide. Whether to expand to the next domain. The pilot fee is credited in full against the first year.
- •Cost. Start with a pilot ($7,500 one-time, credited in full against the first year). Scale is from $1,000 a month and Enterprise from $3,000 a month, billed annually, with unlimited seats, no usage meter and no markup on model spend. See pricing.
The pilot is the start of a longer curve:
Read the full playbook: Data Workers for security and observability data teams