Best practices for agentic public program and government data: a field guide for government data teams
How government data teams move their data back office to agents, practice by practice: the state of the space, the old world and the new, what the agents do step by step, how to set autonomy, who owns what, and how to start.
Agencies must release more data with fewer people.
Profiling, de-identification and documentation are repeatable agent work.
The state of the space
Teams that own program data (claims, enrollment, benefits), web analytics and open data in agencies, often with contractor and digital-service teams. Titles include Chief Data Officer, Data Product Manager and Data Strategy PM.
Agencies are asked to release more data, support AI and protect privacy at once, often with small teams and contractors. Agents can automate the profiling, de-identification, documentation and quality work behind every release.
The old world
A state file changes layout; the monthly release slips a month.
It is rarely one failure. Three patterns repeat:
- •A quarterly program release. Analysts assemble and check each release.
- •Preparing data for ML. A program wants claims data ready for ML.
- •A legacy format change. A state system changes its file layout.
The new world
The layout was mapped on arrival and the release approved on day 3.
Here is that new world as a chain of hand-offs between agents:
These are the impact metrics we move:
- •Access decision: from about 2 weeks to ~1 h
- •Documentation: from about a week to ~1 h
- •Partner layout mapped: from about 3 days to under 1 h
The rest of this report walks through the 5 practices that make that possible. At a glance:
The practices, one by one
1. Run releases as missions
Releases repeat.
Why it is harder than it looks. Each release repeats the same steps under deadline.
The old world. Manual.
The new world. Autopilot.
What the agents do, step by step:
- 1.The Pipeline Building agent calls
generate_pipelineand assembles the release. - 2.The Quality Monitoring agent calls
run_quality_checkand checks it. - 3.The Autonomous Data-Conductor calls
request_governance_reviewand holds for the lead.
In practice. On schedule.
What to measure. Releases on time.
2. De-identify by rule
Consistency protects people.
Why it is harder than it looks. Consistency across files is the real privacy control.
The old world. Per file.
The new world. Data Security.
What the agents do, step by step:
- 1.The Data Security agent calls
scan_piiand classifies. - 2.The Data Security agent calls
review_pii_writesand checks outputs. - 3.The Data Access & Governance agent calls
request_governance_reviewand routes exceptions.
In practice. Exceptions only.
What to measure. Files reviewed by hand.
Practices like these compound. Across the routine work they cover:
3. Map partner layouts
Partners change formats.
Why it is harder than it looks. Partners change layouts on their own schedule.
The old world. Broken loads.
The new world. Schema Evolution.
What the agents do, step by step:
- 1.The Schema Evolution agent calls
detect_schema_changeand detects. - 2.The Schema Evolution agent calls
generate_migrationand maps. - 3.The Pipeline Building agent calls
sandbox_testand tests.
In practice. Day-2 mapping.
What to measure. Broken loads.
4. Generate documentation
Docs lag.
Why it is harder than it looks. Documentation is required and always late.
The old world. Manual.
The new world. generate_documentation.
What the agents do, step by step:
- 1.The Data Context & Catalog agent calls
generate_documentationand writes it. - 2.The Data Context & Catalog agent calls
trace_cross_platform_lineageand adds lineage. - 3.The Data Access & Governance agent calls
generate_audit_reportand packages evidence.
In practice. Current docs.
What to measure. Releases documented.
Autonomy where work is reversible, a named approver where it matters:
5. Grant access by agreement
Requests queue.
Why it is harder than it looks. Access depends on agreements and purpose.
The old world. Tickets.
The new world. provision_access.
What the agents do, step by step:
- 1.The Data Access & Governance agent calls
check_policyand checks agreement. - 2.The Data Access & Governance agent calls
provision_accessand grants. - 3.The Data Access & Governance agent calls
generate_audit_reportand logs.
In practice. Days.
What to measure. Days to access.
Designing the autonomy ladder
Autonomy is set per task class, not per tool. This is where we would set each class for a mature deployment, and why:
| Task class | Closed autonomously | Escalated | Handled by a person | Where we set it | Why |
|---|---|---|---|---|---|
| Documentation | 88% | 8% | 4% | L4 once the receipts are clean | The work is reversible and repeatable |
| Partner mapping | 66% | 24% | 10% | L3: acts and keeps a rollback point | Most cases are routine; the rest escalate with evidence |
| Release checks | 76% | 18% | 6% | L3: acts and keeps a rollback point | Most cases are routine; the rest escalate with evidence |
| De-identification | 36% | 50% | 14% | L3 for routine cases, L2 for the rest | Judgment is needed often enough to keep a person close |
| Releases and identified data | 0% | 86% | 14% | L2: always proposed to a named approver | The number carries accountability outside the data team |
The operating model: who owns what
Agents change who does the work, not who is accountable for it:
| Role | Owns | What the agents take off their plate |
|---|---|---|
| Agency Chief Data Officer | Standards, autonomy levels per domain and which domain goes next | Assembling status, chasing owners, keeping the catalog current |
| Domain owners and approvers | Releases and access to identified data | Preparing the diff, blast radius and rollback plan for every proposal |
| Data engineers | Reviewing agent proposals and building new data products | Releases run as missions with quality and threshold checks; de-identification rules apply consistently |
The swarm behind the practices
| Agent | What it does for you | Tools it calls |
|---|---|---|
| Data Ingestion | Program and partner files | get_connector_status, trigger_airflow_dag, register_kafka_schema |
| Data Security | De-identification and sensitive data | scan_pii, review_pii_writes, pci_card_data_masking |
| Quality Monitoring | Release and threshold checks | set_sla, run_quality_check, get_anomalies, create_quality_tests_for_pipeline |
| Schema Evolution | Legacy and partner formats | detect_schema_change, validate_schema_compatibility, generate_migration, rollback_migration |
| Data Context & Catalog | Documentation and lineage | trace_cross_platform_lineage, blast_radius_analysis, define_business_rule, generate_documentation |
| Data Access & Governance | Access by role and agreement | provision_access, check_policy, request_governance_review, generate_audit_report |
Data Context Wizard keeps one governed graph of definitions, lineage and owners that every agent and person answers from. Autonomous Data-Conductor runs each job from detection to a verified result. Spellbook Data Catalog is where your team approves, audits and rolls back, and asks questions in plain English.
What disappears
- •Slipped releases.
- •Per-file privacy reviews.
- •Broken partner loads.
- •Stale documentation.
- •Access queues.
The magic is not any single agent. It is that the back office keeps running: bad data is caught where it enters, every new table gets quality rules and documentation without anyone writing them, access follows policy, fixes are verified before anyone opens a dashboard, and every action leaves a signed receipt.
A product manager's scorecard
| Metric | How to read it |
|---|---|
| Releases on time | Baseline it in the first week of the pilot; review it monthly. |
| Files reviewed by hand | Baseline it in the first week of the pilot; review it monthly. |
| Broken loads | Baseline it in the first week of the pilot; review it monthly. |
| Releases documented | Baseline it in the first week of the pilot; review it monthly. |
| Days to access | Baseline it in the first week of the pilot; review it monthly. |
Where this goes next
Agencies will publish program data on schedule with privacy and documentation built in, and prepare data for AI programs without large new teams.
The climb: from assisted to autonomous to agentic
Our thesis is that the data platform climbs the way analytics did, in stages, and that the hard part of data work was never knowing what to do but doing it. For government data teams, the climb looks like this:
- •Assisted. Agents work alongside your team. They read the whole stack, show what needs doing and propose every fix with a receipt; releases and access to identified data always go to a named approver.
- •Autonomous, under your governance. Routine work runs on its own: releases run as missions with quality and threshold checks, de-identification rules apply consistently, and documentation is generated as data changes. Your team governs the agents instead of doing the upkeep.
- •The agentic enterprise. Agencies will publish program data on schedule with privacy and documentation built in, and prepare data for AI programs without large new teams.
You choose the altitude, every step is visible and reversible, and the pilot below is the first rung.
The pilot: how government data teams start
The pilot is how your team proves the value on its own data before committing.
- •What we connect. Government cloud environments; Databricks or Snowflake; SAS and legacy systems; open data portals; analytics for public websites, over MCP, inside your infrastructure. Nothing migrates.
- •Scope. One monthly release at L2.
- •First week. Agents run at L1 across the scoped domain. They map every pipeline, definition and lineage path and show the fixes they would have made, with receipts, so your team sees the value before anything changes.
- •By week two. Releases run as missions with quality and threshold checks and de-identification rules apply consistently start running on reversible work. Releases and access to identified data stay at L2 with a named approver.
- •At day 30. We review the impact metrics against your baseline:
- •Access decision: from about 2 weeks to ~1 h
- •Documentation: from about a week to ~1 h
- •Partner layout mapped: from about 3 days to under 1 h
- •the share of documentation closed autonomously, targeting 88% as the deployment matures
- •What you decide. Whether to expand to the next domain. The pilot fee is credited in full against the first year.
- •Cost. Start with a pilot ($7,500 one-time, credited in full against the first year). Scale is from $1,000 a month and Enterprise from $3,000 a month, billed annually, with unlimited seats, no usage meter and no markup on model spend. See pricing.
The pilot is the start of a longer curve:
Read the full playbook: Data Workers for government data teams