A playbook for Clinical Data Product Managers
If you are a Clinical Data Product Manager, this is your playbook. It covers what your team ships, where its week goes today, the use cases agents take over first, the autonomy you can safely grant in each domain, and what to show leadership after the first quarter.
Key takeaways
- •Who this is for. Teams that turn EHR data into products: health-system research warehouses on Clarity and Caboodle, enterprise data and analytics groups inside health IT, clinical data vendors, and real-world data companies that aggregate across sites.
- •The problem today. Quarter-long partner onboarding, study-by-study PHI review and a cohort request queue measured in weeks.
- •What changes. Every new field is classified for PHI the moment it lands, so privacy review stops being the last, slowest step of every project. Nightly loads are verified and reruns handled automatically, so researchers stop discovering missing encounters in their results.
- •What stays with your people. New code mappings and access to identified data: patient data and research validity need a clinician's or IRB-approved person's sign-off.
- •The result. Clinical informaticists spending their time on the mappings that need clinical judgment, and researchers getting cohorts in days.
The PRD for agentic clinical and EHR data teams
This is the requirements document we would write with you for your team. Everything below expands on it.
| Who | Health-system data warehouses, clinical data vendors, research informatics and real-world data teams |
| The job | New partner sites live in a week. |
| Environment | Epic Clarity and Caboodle; HL7 and FHIR feeds via an integration engine; Snowflake, Databricks or a research warehouse; OMOP; dbt. |
| Constraints | New code mappings and access to identified data always go to a named approver. HIPAA requires minimum-necessary access and safeguards for PHI, and research use runs through IRB approvals. |
| Success is measured by | PHI classification on a new feed: from about a week to minutes; Finding the right dataset: from about 8 hours to 30 s; Local lab code mapped: from about 2 days to under 1 h; data dictionary upkeep closed autonomously 86% of the time. |
| Owned by | A Clinical Data Product Manager, with domain owners as named approvers. |
| Pilot scope | One partner site onboarded as a mission. |
What going fully agentic and autonomous means for clinical and EHR data teams
- •Put your data pipeline on autopilot. Every new field is classified for PHI the moment it lands, so privacy review stops being the last, slowest step of every project.
- •Run the data back office without the busywork. The work that fills the week today moves to agents: quarter-long partner onboarding, study-by-study PHI review and a cohort request queue measured in weeks.
- •Scale data without scaling headcount. Nightly loads are verified and reruns handled automatically and the data dictionary and lineage update as the agents work, so the data you manage grows without the team having to.
- •Get the plumbing off your plate. Clinical informaticists spending their time on the mappings that need clinical judgment, and researchers getting cohorts in days.
Who this is for
Teams that turn EHR data into products: health-system research warehouses on Clarity and Caboodle, enterprise data and analytics groups inside health IT, clinical data vendors, and real-world data companies that aggregate across sites. Titles include Clinical Data Product Manager, Data Product Owner, Data Foundations Director and Research Informatics Lead.
The product you ship
- •Research and QI datasets with identified and de-identified versions.
- •Cohorts researchers and operations teams can query in hours, not months.
- •Standardized models such as OMOP that make data comparable across sites.
- •Licensed real-world datasets for life-sciences customers.
Where the week goes
- •Every site is different. New partners send ADT, HL7 and FHIR with local codes and local habits.
- •Mappings never finish. Local lab and medication codes need mapping to LOINC, RxNorm and SNOMED.
- •PHI makes everything slower. Every new table needs classification, access rules and review before anyone uses it.
- •Request queues grow. Researchers wait weeks for a cohort extract.
- •Documentation lags. The data dictionary is always out of date.
Add it up and upkeep is most of the team's week. Here is how a typical week splits today, and once agents own the back office:
Time spent on new data products moves from 14% of the week to 50%. Most of the rest becomes review: people steer and approve agent work instead of doing it by hand.
Then and now
| Work | Today | With Data Workers |
|---|---|---|
| A new partner site | One to three months | About a week |
| Local code mapping | A permanent backlog | Mapped with confidence scores; experts review only the uncertain |
| PHI review | At the end of each project | On arrival, before anyone opens a row |
| Cohort requests | Weeks in a queue | Days, from governed concepts |
| Data dictionary | Out of date | Kept current by agents |
| OMOP version upgrades | A multi-quarter project | A mission with parity checks |
These are the impact metrics we move for clinical and EHR data teams:
- •PHI classification on a new feed: from about a week to minutes
- •Finding the right dataset: from about 8 hours to 30 s
- •Local lab code mapped: from about 2 days to under 1 h
Across the routine work in this domain:
The biggest change is PHI classification on a new feed: from about a week to minutes. Work that waited in queues finishes in minutes or hours, because agents detect, fix and verify in one pass.
Use cases, in depth
A partner site live in a week
Today. The integration engineer, the privacy reviewer and the mapping analyst each take weeks in sequence.
With Data Workers. Seven agents run connect, land, classify, govern, model, test and document as one mission; one clinician reviews uncertain lab codes.
What changes. The research network grows without growing the team.
A cohort before the grant deadline
Today. The request waits behind 30 others and is written by hand from scratch.
With Data Workers. Context Wizard supplies the governed concept, Access & Governance checks the IRB approval, and Pipeline Building produces the extract from tested models.
What changes. Researchers get cohorts in days, defined the same way every time.
Moving the warehouse to a new OMOP version
Today. A large project that freezes other work.
With Data Workers. Data Migration moves tables in waves with parity checks, Quality Monitoring confirms counts by domain, and documentation updates as each wave lands.
What changes. The upgrade happens alongside normal work.
Agents working together: A new partner hospital, live in a week
New sources are where data teams lose quarters. A new hospital partner, a new payer file, a new 3PL: every one needs a connection, a landing zone, profiling, privacy review, access rules, models, tests, documentation and a freshness SLA. Each step waits on the last. Data Workers runs them as one onboarding mission.
The mission
A clinical data team adds a new health-system partner. The partner sends ADT messages over HL7 and a nightly FHIR bulk export of encounters, labs and medications. The team lead types one outcome into Autopilot:
Onboard Riverside Health's ADT and FHIR feeds into the encounters, labs and meds domains, PHI-safe, with a 6am freshness SLA.
Autopilot breaks that into work for the swarm, shows the estimated token cost, and starts. New sources start at L1 for anything touching PHI; everything else runs at the levels the team already set.
Step 1 — connect
Connectors agent sets up the HL7 listener and the FHIR bulk export pull using the credentials the partner provided through the team's secrets manager. It tests both endpoints and records the connection in the catalog.
Step 2 — land and profile
Data Ingestion agent lands the first export in a quarantined raw zone and profiles every resource type: row counts, null rates, code systems in use, and fields that do not match the team's existing data model. It finds that this partner sends lab codes in a local system as well as LOINC.
Step 3 — find the PHI before anyone else does
Data Security agent classifies every field. Names, MRNs, dates of birth, addresses and free-text notes are tagged as PHI. Sensitive values are scrubbed before anything is written to the context graph. Nobody on the team has had to open a raw row.
Step 4 — apply access policy
Data Access & Governance agent applies the team's existing policies to the new tables: raw PHI to the restricted role, de-identified views to analysts, nothing to the sandbox. It opens access for the two engineers already on the partner project and logs every grant.
Step 5 — build the models
Pipeline Building agent maps the partner's encounter, lab and medication resources into the team's canonical model. It writes the local-to-LOINC lab mapping as a reviewed seed table, builds the dbt models and tests, and flags the 3% of lab codes it could not map with confidence for a clinical informaticist.
Step 6 — make it trustworthy
Quality Monitoring agent writes rules for the new source: encounter volume by facility, lab result ranges, admit-discharge ordering, and duplicate MRNs. Data Context & Catalog agent documents every table, column and mapping, and links lineage from the HL7 message to the cohort table researchers use.
Step 7 — keep it running
Autonomous Data-Conductor sets the 6am freshness SLA, watches the first week of loads, and handles the reruns itself when the partner's export lands late on Tuesday. It escalates exactly one decision all week: the unmapped lab codes.
Friday
The feed is live. Researchers query encounters from the new partner through the same governed definitions as every other site, the catalog page shows owner, SLA, schema history and PHI tags, and every action in the onboarding has a receipt.
What changed for the team
- •One request instead of a project plan. The outcome was stated once. The swarm did the sequencing.
- •PHI handled first, not last. Classification and policy happened before models were built, not as a review at the end.
- •The expert time went where it mattered. The only human work was the clinical mapping decision that genuinely needed a clinician.
The same mission runs for a payer eligibility file, a 3PL ship-notice feed or a new ad partner. Only the domain changes.
Minute by minute
This is an illustrative example. A partner health system sends ADT over HL7 and nightly FHIR bulk exports; the research warehouse runs OMOP on Databricks. PHI-touching work starts at L1.
| Time | Who | What happens |
|---|---|---|
| Mon 09:00 | Autopilot | Onboarding mission stated; token estimate shown |
| 09:20 | Data Ingestion agent | Connectors set up the HL7 listener and FHIR pull |
| 11:00 | Data Ingestion agent | First export profiled; local lab codes found |
| 11:30 | Data Security agent | scan_pii classifies PHI before anyone opens a row |
| Tue | Data Access & Governance agent | provision_access applies IRB and role policy |
| Wed | Pipeline Building agent | OMOP mappings with tests; 3% of lab codes flagged |
| Thu | Spellbook Data Catalog | Clinician reviews uncertain codes |
| Fri | Research warehouse | Partner live; lineage documented |
The agents on the job
| Agent | What it does for clinical data teams | Tools it calls |
|---|---|---|
| Connectors and Data Ingestion | Land ADT, HL7, FHIR and Clarity extracts, and profile every load | get_connector_status, trigger_airflow_dag, get_dbt_run_history |
| Data Security | Classifies PHI field by field and scrubs it before it reaches context | scan_pii, review_pii_writes, pci_card_data_masking |
| Data Access & Governance | Applies IRB and role policies to every table and logs every grant | provision_access, check_policy, request_governance_review, generate_audit_report |
| Pipeline Building | Maps sources into OMOP or your canonical model and writes tests | generate_pipeline, validate_pipeline, sandbox_test, deploy_pipeline |
| Quality Monitoring | Encounter volume, lab range, ordering and duplicate-patient checks | set_sla, run_quality_check, get_anomalies, create_quality_tests_for_pipeline |
| Data Context & Catalog | Keeps the data dictionary and lineage current, from message to cohort | trace_cross_platform_lineage, blast_radius_analysis, define_business_rule, generate_documentation |
Context Wizard holds governed concept definitions, such as a diabetic patient or a readmission, so cohorts mean the same thing in every study. Conductor keeps nightly loads fresh and verified. Spellbook lets researchers find datasets and request access in plain English.
What goes on autopilot
Once a domain has earned it, this work runs on its own, and each item removes a problem your team deals with today:
- •Every new field is classified for PHI the moment it lands, so privacy review stops being the last, slowest step of every project.
- •Nightly loads are verified and reruns handled automatically, so researchers stop discovering missing encounters in their results.
- •The data dictionary and lineage update as the agents work, so documentation is finally current.
Where your people stay in charge
Agents never get blanket access. Each domain has its own autonomy level, set by you, and it only rises when the receipts show the agents are reliable. Everything starts at L1, observing only; reversible upkeep such as reruns, staging fixes, documentation and cleanup is usually first to run on its own.
- •New code mappings and access to identified data always goes to a named approver: patient data and research validity need a clinician's or IRB-approved person's sign-off.
- •Clinical judgment and IRB decisions: these never move to agents.
Once a deployment has matured, here is how that ladder plays out across task classes:
Data dictionary upkeep closes autonomously 86% of the time. For Identified-data access, autonomous closure is 0% by design: every change is proposed with a full diff and approved by a named person.
What happens when an agent is wrong
- •At L2 propose, nothing changes until a named person approves. The proposal shows the full diff, the blast radius and how to roll it back. Unanswered requests expire and escalate.
- •At L3 act, reversibly, every change keeps a rollback point, and Conductor verifies the downstream number after the change. If verification fails, the change is reversed and the domain drops a level.
- •No agent approves or promotes its own work. Autonomy only grows on evidence, per repair class, and repeated errors shrink scope automatically.
- •One org-wide stop halts all autonomous dispatch, and every action lands in a hash-chained audit trail you can export with generate_audit_report.
How it fits your stack
Data Workers connects to what you already run: Epic Clarity and Caboodle; HL7 and FHIR feeds via an integration engine; Snowflake, Databricks or a research warehouse; OMOP; dbt. Each agent is an MCP server, so the same agents work inside Claude Code, Cursor or Codex and in Slack or Teams through Spellbook. Nothing migrates.
- •Runs in your infrastructure with your credentials and your model key. The hosted Autonomous Data-Conductor sees workflow metadata only, never rows.
- •Uses the access you already govern. Agents act under scoped service identities and your existing roles; every grant goes through provision_access and is logged.
- •Proposes changes as diffs in your dbt project, orchestrator and warehouse, with blast radius, tests and a rollback point, and applies them only at the autonomy level you set for that domain.
- •Open-source core. The Apache-2.0 community edition ships 11 agents and 160+ MCP tools, so your engineers can read exactly what the agents call.
Governance and regulation
HIPAA requires minimum-necessary access and safeguards for PHI, and research use runs through IRB approvals. Data Workers classifies PHI on arrival, scrubs sensitive values before anything reaches context, applies role and IRB policy and logs every access.
Data Workers doesn't make you compliant; your controls, people and assessors do. It supplies controls and evidence wherever agents touch data. Not legal advice.
The alternatives clinical and EHR data teams weigh
Build it yourselves. A strong team can wire a coding agent to MCP servers for the warehouse and dbt. The work is everything around it: routing approvals to the right owner, blast radius, rollback, receipts, and current context about every other system.
Observability, catalogs and point tools. Observability alerts, catalogs document, and quality tools test. Data Workers takes their signals, diagnoses, proposes the fix, verifies it and leaves the receipt.
Platform-native agents. Snowflake made its managed MCP server and Cortex Agents generally available on Nov 4, 2025, and Databricks made Genie One generally available on Jun 16, 2026. They are excellent inside their platform; this work crosses the source systems, orchestrator, warehouse and BI, and Data Workers works across all of them.
In this space specifically. Integration engines move messages and research platforms host data; Data Workers does the onboarding, mapping, PHI handling and documentation between them.
The case for your CTO
What this replaces. Quarter-long partner onboarding, study-by-study PHI review and a cohort request queue measured in weeks.
What your team gets back. Clinical informaticists spending their time on the mappings that need clinical judgment, and researchers getting cohorts in days.
Integration effort. Connecting is configuration, not a migration. The agents attach over MCP to the warehouse, orchestrator, dbt project and source systems you already run, under the roles you already govern, inside your own infrastructure. Your engineers can read every tool the agents call in the open-source core.
Risk and control. Autonomy is set per domain. Everything starts at L1, observing only; reversible work moves up only when the receipts justify it; new code mappings and access to identified data stay at L2 with a named approver; and one org-wide stop halts all autonomous work.
Why now. Cosmos, research networks and AI pilots multiply data requests.
The first win. One partner site onboarded as a mission. The pilot below lays out the scope, what you see each week and the cost.
The climb: from assisted to autonomous to agentic
Our thesis is that the data platform climbs the way analytics did, in stages, and that the hard part of data work was never knowing what to do but doing it. For clinical and EHR data teams, the climb looks like this:
- •Assisted. Agents work alongside your team. They read the whole stack, show what needs doing and propose every fix with a receipt; new code mappings and access to identified data always go to a named approver.
- •Autonomous, under your governance. Routine work runs on its own: every new field is classified for PHI the moment it lands, nightly loads are verified and reruns handled automatically, and the data dictionary and lineage update as the agents work. Your team governs the agents instead of doing the upkeep.
- •The agentic enterprise. Clinical informaticists spending their time on the mappings that need clinical judgment, and researchers getting cohorts in days. People set direction, and the platform carries it out.
You choose the altitude, every step is visible and reversible, and the pilot below is the first rung.
The pilot: how clinical and EHR data teams start
The pilot is how your team proves the value on its own data before committing.
- •What we connect. Epic Clarity and Caboodle; HL7 and FHIR feeds via an integration engine; Snowflake, Databricks or a research warehouse; OMOP; dbt, over MCP, inside your infrastructure. Nothing migrates.
- •Scope. One partner site onboarded as a mission.
- •First week. Agents run at L1 across the scoped domain. They map every pipeline, definition and lineage path and show the fixes they would have made, with receipts, so your team sees the value before anything changes.
- •By week two. Every new field is classified for PHI the moment it lands and nightly loads are verified and reruns handled automatically start running on reversible work. New code mappings and access to identified data stay at L2 with a named approver.
- •At day 30. We review the impact metrics against your baseline:
- •PHI classification on a new feed: from about a week to minutes
- •Finding the right dataset: from about 8 hours to 30 s
- •Local lab code mapped: from about 2 days to under 1 h
- •the share of data dictionary upkeep closed autonomously, targeting 86% as the deployment matures
- •What you decide. Whether to expand to the next domain. The pilot fee is credited in full against the first year.
- •Cost. Start with a pilot ($7,500 one-time, credited in full against the first year). Scale is from $1,000 a month and Enterprise from $3,000 a month, billed annually, with unlimited seats, no usage meter and no markup on model spend. See pricing.
The pilot is the start of a longer curve:
By month 12, more than 34% of agent actions run fully autonomously and manual work falls below 10%, while identified-data access stay with a named approver.
FAQ
Does PHI leave our environment? No. Agents run in your infrastructure and sensitive values are scrubbed before context.
Does it work with Epic? Through Clarity, Caboodle and your feeds.
Who approves code mappings? A clinician or informaticist for anything uncertain.
Where do the agents run? In your infrastructure, with your credentials and model key. The hosted Conductor sees workflow metadata only.
Do we have to migrate anything? No. Your warehouse, dbt project, orchestrator, catalog and BI stay where they are.
Sources
- •Mount Sinai Data Warehouse: https://labs.icahn.mssm.edu/msdw/about-us/
- •Fierce Healthcare, Epic Cosmos and Curiosity: https://www.fiercehealthcare.com/health-tech/epic-expands-ai-ambitions-agent-platform-cosmos-powered-predictions-and-deeper-workflow
- •Flatiron Health, Data quality 2.0: https://resources.flatiron.com/real-world-evidence/data-quality-real-world-evidence
- •Data Workers pricing: https://dataworkers.io/pricing/
- •Data Workers open-source repository (tool registrations): https://github.com/DataWorkersProject/dataworkers-claw-community
- •Snowflake, managed MCP server and Cortex Agents GA release note (Nov 4, 2025): https://docs.snowflake.com/en/release-notes/2025/other/2025-11-04-cortex-agents-mcp
- •Databricks, Genie One launch (Jun 16, 2026): https://www.databricks.com/company/newsroom/press-releases/databricks-launches-genie-one-all-new-agentic-coworker-every-team