Best practices for agentic genomics, imaging and multi-omics data: a field guide for genomics data teams

Ask a bioinformatics data lead what went wrong last quarter and the answer is rarely the science. A collaborating site sent a sample sheet with its own tissue terms. An annotation release was upgraded on a Friday and a gene panel's carrier counts moved. A reviewer asked for the exact participants behind a figure from two years ago, and the answer lived in three notebooks and an email thread. The team that owns the metadata and tabular layer around sequencing, imaging and proteomics absorbs every change on both sides.

The mandate is reusable data: cohorts a scientist can start from, results traced to the pipeline that produced them, and consent visible on every sample. The trap is that, without automation, the team spends its week reconciling spreadsheets and rebuilding last year's analysis.

Our view, as people who have run data platforms: the work that keeps this layer correct is repeatable, and agents should own it. Analysis stays where it runs, and consent and release stay with people. Trial data has its own field guide for clinical and R&D data, and the healthcare and life sciences platform guide covers the wider estate. This guide is for health and life sciences data teams and is the practitioner's companion to the genomics, imaging and multi-omics playbook.

The old world

It is 9:00 on a Monday. Over the weekend the core upgraded an annotation database release and re-ran batch 14. Every job reported success and nobody was paged. On Thursday a scientist notices that carrier counts in a gene panel no longer match last quarter's slide, and spends two days working out whether it is a sample swap, a failed load or the release. It is the release, and the number has been in a draft figure and two exploratory cohorts for a week.

Meanwhile a new hospital site's sample sheet arrives in its own layout with a free-text consent column, and an engineer maps it by hand. A partner has asked for a cohort, and the consent evidence is being assembled from a spreadsheet and a PDF.

The new world

Same Monday with Data Workers. The Connectors agent sees the annotation release changed. Quality Monitoring compares the new results with what it has learned is normal for each table: 212 variant calls changed class in three panels, while sample counts and key uniqueness are unchanged, so the load is sound. Data Context & Catalog lists the two cohorts, the figure and the submission table that read them, and Incident Debugging attributes the shift to the release. Autonomous Data-Conductor sends the bioinformatics lead one decision with the full diff and a proposal to pin the submission to the old release. The new site's sample sheet was mapped on arrival, with the columns that did not fit listed for a curator.

Here is that Monday as a chain of hand-offs between agents:

Seven practices make that Monday possible. At a glance:

The practices, one by one

1. Map every sample sheet and assay export the day it arrives

Why it's harder than it looks: layouts are not versioned, the same field has three names across three sites, and the mapping script is a one-off that only one person understands. When that person leaves, the next sheet takes a week.

1. 1.The Data Ingestion agent profiles each arriving file for columns, value sets, duplicates and missing keys.
2. 2.The Schema Evolution agent calls detect_schema_change and validate_schema_compatibility against the sample model and lists the columns that do not fit.
3. 3.It calls generate_migration for the mapping, with a rollback plan, so a layout change is a reviewed diff and not a script.
4. 4.A curator approves the mappings that are not obvious, such as a free-text field that probably encodes sample type.

2. Pin every result to the pipeline and reference version that made it

Nothing breaks when an annotation or reference release changes. The numbers just differ, and a result that carries no version can only be argued about.

Why it's harder than it looks: the workflow manager records the version, the results table does not, and a rerun overwrites the earlier release. By the time someone asks why a count moved, the baseline is gone.

1. 1.The Connectors agent reads run history and version tags on every load with get_dbt_run_history.
2. 2.The Quality Monitoring agent calls run_quality_check and get_anomalies to compare each table with its prior release: class counts, distributions, sample counts, null rates and key uniqueness.
3. 3.The Data Context & Catalog agent calls blast_radius_analysis to list what reads it, and Incident Debugging calls get_root_cause to separate a release change from a sample swap or a failed load.
4. 4.Pipeline Building proposes version-pinned tables, with a test that fails if a submission table reads an unpinned version. The bioinformatics lead decides which analyses stay on the old release.

Across the routine work in this area, here is the difference in elapsed time:

3. Hold one governed definition for every ontology term and cohort criterion

A mapping that drifts quietly changes who is in a cohort.

Why it's harder than it looks: mappings are fixed per study by whoever was curating that month. A new HPO, SNOMED CT or UBERON release moves a term, and nothing tells the cohorts that used the old one.

1. 1.The Data Context & Catalog agent calls define_business_rule for each confirmed phenotype, tissue, assay and inclusion criterion, and mark_authoritative for the table that holds the trusted mapping.
2. 2.Data Context Wizard proposes mappings for incoming terms with provenance, and get_context returns the governed definition to scientists and AI assistants alike.
3. 3.When a release or mapping changes, blast_radius_analysis lists every cohort definition that would move, and a named curator approves any change that alters one.

Keep the catalog you already run. See how an agentic data catalog stays in step with the data, and the self-maintaining catalog transformation.

4. Define cohorts as versioned models you can rebuild as published

A cohort is a claim: these participants, these tables, these versions, this consent. As a notebook, it cannot be checked later.

Why it's harder than it looks: the inclusion rules sit across a notebook, an email and a figure legend, and eighteen months on the tables have been re-run twice.

1. 1.The Pipeline Building agent calls generate_pipeline to express the cohort as a model in your dbt project, then validate_pipeline.
2. 2.Quality Monitoring calls create_quality_tests_for_pipeline for duplicate participants, missing consent flags and impossible values.
3. 3.Data Context & Catalog calls trace_cross_platform_lineage and generate_documentation, so sources, versions and methods are written down as the cohort is built.
4. 4.The scientist who owns the analysis approves the definition, and the model is versioned like any other code.

Rebuilding is then a run, not a reconstruction. See automated data lineage and the end-to-end lineage transformation.

5. Attach consent and data-use terms to the data, and apply them as filters

Consent differs per participant, and each dataset carries its own data-use terms. If those live in a PDF, every cohort relies on someone remembering them.

Why it's harder than it looks: consent arrives as free text in site-specific wording, and withdrawals come after the cohort was built. The GA4GH Data Use Ontology gives a shared vocabulary for dataset permissions, but someone still has to carry the codes next to the data.

1. 1.The Data Ingestion agent lists samples with a missing, ambiguous or conflicting consent value on arrival.
2. 2.Data Access & Governance calls check_policy against the recorded terms for each dataset and cohort, and enforce_rbac to scope participant-level tables to the roles that need them.
3. 3.Quality Monitoring adds a test through create_quality_tests_for_pipeline that fails if a cohort holds a participant whose consent does not cover the use.
4. 4.The data access committee decides what a consent value means and whether a use is permitted. Agents flag conflicts and never widen a permitted use; approved requests go through provision_access and are logged. See zero-ticket data access.

6. Screen imaging and header metadata for identifiers before it spreads

Identifiers hide in DICOM header fields, private tags and burned-in text, and metadata extracted to a table is easy to query and just as easy to leak.

Why it's harder than it looks: upstream de-identification is assumed to have worked and nobody checks the extracted tables. The DICOM standard defines the fields, not which ones your sites filled with names. Agents see extracted metadata, never pixels, so burned-in text stays with your imaging tools.

1. 1.The Data Ingestion agent profiles each extracted header table for populated, free-text and vendor-specific fields.
2. 2.Data Access & Governance calls scan_pii on the tables you point it at, by name and by content, without sending sample values outside your network.
3. 3.Data Context & Catalog calls auto_tag_dataset, so access, lineage and quality treat tagged columns differently from then on, and a handling decision is proposed for each. See data classification.
4. 4.The imaging informatics lead approves the decision before the table reaches analysts.

7. Keep consent scope and cohort release with the committee, and hand it the evidence

Why it's harder than it looks: evidence is assembled late and by hand, from lineage in one tool, consent in another and the agreement in a PDF, so committees wait weeks or approve on a summary nobody can check.

1. 1.Data Context & Catalog calls trace_cross_platform_lineage for the requested cohort and generate_documentation for its data dictionary.
2. 2.Data Access & Governance calls check_policy against the recorded terms and request_governance_review to route the release, with the consent filter applied and shown.
3. 3.It calls generate_audit_report, so the evidence pack exports from the tamper-evident audit trail.
4. 4.The named owner on the data access committee approves, narrows or refuses the release. Unanswered requests expire and escalate.

Once a deployment has matured, here is how that ladder plays out across task classes:

Designing the autonomy ladder

Autonomy is set per task class, and only a domain's owner can raise it. Everything starts at L1, observing only. L0 to L2 run today: watch, suggest, or act after approval. L3 and L4 are where a domain can go on evidence from its own receipts, never by default:

The operating model: who owns what

Agents change who does the work, not who is accountable for it:

The swarm behind the practices

No single agent delivers these practices. Each one is a hand-off between specialists, coordinated under the autonomy you set per domain.

Data Context Wizard holds one governed definition of each phenotype, tissue, assay and cohort criterion, so every analysis and AI assistant answers from the same terms. Autonomous Data-Conductor verifies the outcomes that matter before a cohort is used or released. Spellbook Data Catalog (in preview) gives each cohort a page that updates when the data does. For the regulatory side, see how Data Workers helps with HIPAA, GDPR and the EU AI Act.

What disappears

• •The silent shift found by a reviewer after a release upgrade.
• •Mapping a collaborator's sample sheet by hand.
• •The same tissue under three spellings and two ontology releases.
• •A cohort that cannot be rebuilt because its notebook moved on.
• •Consent evidence assembled the night before a committee meeting.

The magic is not any single agent. It is that the metadata layer keeps running: changes are caught where they enter, not where a scientist reads them, and every action leaves a signed receipt, so the team can see what happened and roll anything back.

A genomics data leader's scorecard

If you own this layer as a product, these are the targets to bring to every quarterly review:

• •Time to rebuild a published cohort, targeting hours and not weeks.
• •Time from a pipeline or reference change to a traced result shift, targeting minutes.
• •Cohorts in papers or submissions with versioned definitions, lineage and consent basis attached, targeting all of them.
• •External releases with a named approver and an evidence pack on record, targeting every one.
• •Catalog and results-table documentation closed autonomously, and its token cost, targeting 85% as the deployment matures.
• •Analyses redone because of a data surprise, targeting a steady fall.

The climb: from assisted to autonomous to agentic

Our thesis is that the data platform climbs the way analytics did, in stages, and that the hard part of data work was never knowing what to do but doing it. For genomics, imaging and multi-omics data teams, the climb looks like this:

• •Assisted. Agents read the whole stack, show what needs doing and propose every fix with a receipt; consent scope, data-use agreements and cohort release always go to a named owner.
• •Autonomous, under your governance. Routine work runs on its own: sample sheets are mapped on arrival, results are compared with their prior release whenever a version changes, and data dictionaries stay current.
• •The agentic enterprise. A genomics data team that ships reusable cohorts and clean results tables instead of reconciling spreadsheets. People set direction, and the platform carries it out.

You choose the altitude, every step is visible and reversible, and the pilot below is the first rung.

The pilot: how genomics, imaging and multi-omics data teams start

The pilot is how your team proves the value on its own data before committing.

• •What we connect. The warehouse or lakehouse holding one assay family's results, the sample metadata, dbt, the orchestrator and your catalog, over MCP, inside your infrastructure. Nothing migrates, and raw files stay in object storage.
• •Scope. Pipeline-version drift checks and cohort documentation on one assay family, at L1 and L2.
• •First week. Agents run at L1. They map every table, definition and lineage path and show the version comparisons and cohort documentation they would have produced, with receipts, before anything changes.
• •By week two. Sample sheet mapping and docs upkeep start running on reversible work. Results-table changes and cohort definitions stay at L2 with a named approver; consent scope and release stay with the committee.
• •At day 30. We review the impact metrics against your baseline:

• •Cohort rebuilt as published: from about 40 hours to 2 hours
• •Result shift traced to a pipeline change: from about 24 hours to 30 minutes
• •Consent evidence for a release assembled: from about 12 hours to 1 hour
• •the share of catalog and docs upkeep closed autonomously, targeting 85% as the deployment matures

• •Guarantee. In a 90-day pilot, ten engineering hours per data engineer per week recovered against a baseline both sides sign on day one, or the $7,500 fee is refunded.
• •What you decide. Whether to expand to the next assay family or biobank. The pilot fee is credited in full against the first year.
• •Cost. Start with a pilot ($7,500 one-time, a 6 to 12 week pilot, credited in full against the first year). Scale is from $1,000 a month and Enterprise from $3,000 a month, billed annually, with unlimited seats, no usage meter and no markup on model spend. See pricing.

The pilot is the start of a longer curve:

By month 12, around 38% of agent actions run fully autonomously and manual work falls well below where it starts, while consent scope, data-use terms and external release stay with a named owner.