Best practices for agentic data acquisition and sourcing: a field guide for data sourcing teams

If your company sells data, the product is only as good as the weakest supplier behind it. Customers compare data vendors on coverage, accuracy and freshness, and all three are set upstream: by which sources you license, how fast each goes live and how quickly you notice when one slips. A thin file becomes a stale feed, and a stale feed becomes an incident, with a service credit if the contract says so.

The mandate for a Head of Data Acquisition is commercial: raise coverage and freshness per dollar of licensing and engineering. The trap is that the work around each supplier (landing, profiling, mapping a layout change, building the overlap spreadsheet, finding the licence clause) grows with the number of sources while the team does not.

Our view: that work is repeatable, so agents should do it, and people should keep the decisions that carry money and legal exposure. This field guide, the companion to the team playbook, follows a source through its life and judges each practice by what it does to the data you sell.

The old world

It is 07:30 on a Monday. A supplier's weekly panel file landed on time and in the right layout, so nothing failed. It is a third smaller than usual, because the supplier lost a data partner and two of nine regions are missing. The build ran and the feed went out to customers at 08:00. On Wednesday a customer asks why two regions fell off a cliff. An engineer finds the cause by comparing row counts by hand, and legal checks whether anyone is owed a credit. Meanwhile three suppliers cover overlapping companies, one renews next month, and the only comparison on file is the account manager's deck.

The new world

Same Monday with Data Workers. The Connectors agent logged the delivery, Quality Monitoring flagged the drop in rows and fill rate, and Incident Debugging traced it to two regions absent upstream. The Data Context & Catalog agent listed the feeds that read the file. Autonomous Data-Conductor held the build for those regions and sent the sourcing lead one decision: serve last week's vintage with a coverage note, or publish as is. She approved it at 07:00 and no customer found the gap.

The rest of this report walks through the 7 practices that make that morning possible. At a glance:

The practices, one by one

1. Run every source as a governed record, not a folder

You can list your suppliers from memory and still not know, for any one, the contract end date, what the licence allows and which products read it.

Why it's harder than it looks: the facts live in five places: legal holds the contract, the orchestrator the schedule, finance the cost, dbt the lineage and someone's head the owner. The spreadsheet that merges them is wrong within a quarter.

1. 1.The Connectors agent calls get_connector_status to record the feed, its schedule and its last delivery.
2. 2.The Data Context & Catalog agent runs generate_documentation for the source page (owner, contract dates, cost, the licence terms your team enters) and trace_cross_platform_lineage to attach the products that read it.
3. 3.Quality Monitoring sets the freshness you promise customers with set_sla, so the source carries the same number as the product.
4. 4.The sourcing lead approves the record and any later change to owner or terms.

2. Test the first full delivery against the trial sample

The sample is curated by the supplier. The first full delivery is the one that matters, and it is often thinner, later or narrower in geography.

Why it's harder than it looks: by then the contract is signed and the team is on the next deal. The comparison happens only if someone remembers, so the gap surfaces as a customer complaint.

1. 1.Quality Monitoring runs run_quality_check on the trial sample, profiling fill rate, regional coverage, code lists and date formats, and keeps that profile as the baseline.
2. 2.The Data Ingestion agent lands every delivery the same way, so the first full file is compared with the sample automatically.
3. 3.The agents report where it departs (fewer regions, lower fill rate, a different code list) and create_quality_tests_for_pipeline turns each difference into a check.
4. 4.The data partnerships manager decides whether to proceed, renegotiate or walk away.

Practices like these compound. Across the routine work they cover, here is the difference in elapsed time:

3. Land each supplier once, then normalise once

Suppliers send odd-delimiter CSV, nested JSON, zipped SFTP drops, quarterly Parquet and APIs with their own pagination. The temptation is one loader per supplier.

Why it's harder than it looks: the loader is the easy part. The cost is in the second and third delivery, when a file arrives twice, partially or with a header moved. A raw landing zone that survives re-runs, plus a normalisation layer your products read, keeps a bad delivery from reaching a customer. See the Data Ingestion agent, and the custom connector build for suppliers with no connector.

1. 1.The Connectors agent registers the source and, where none exists, builds a connector from the supplier's API documentation or file format.
2. 2.Pipeline Building calls generate_pipeline for the landing and normalisation models and validate_pipeline to test them against the sample, as a reviewed change in dbt.
3. 3.deploy_pipeline applies the change at the autonomy level you set, and a data engineer approves the first deployment for each new supplier.

4. Write each supplier's expectations as a contract

Most supplier relationships have a commercial contract and no data contract: nothing records the columns, delivery time, minimum fill rate or code lists, so a delivery is checked against memory.

Why it's harder than it looks: expectations differ per supplier and shift at renewal, and writing them by hand for fifty suppliers is a project nobody funds. An open format such as the Open Data Contract Standard keeps it readable by people and agents; see how agents enforce data contracts.

1. 1.From the first deliveries, Quality Monitoring drafts the expectations: layout, freshness and completeness.
2. 2.The Data Context & Catalog agent records them against the source with define_business_rule, next to the licence terms.
3. 3.On every delivery run_quality_check compares the file with the contract and check_freshness compares arrival with schedule (freshness monitoring and SLAs).
4. 4.Each miss lands on the supplier's scorecard; the sourcing lead decides whether to raise it or change the expectation.

5. Map layout changes the day they happen

Suppliers rename columns, split fields and add code values without telling you. Nothing fails; the product just gets quietly worse.

Why it's harder than it looks: a changed layout rarely breaks ingestion. It loads nulls, and fill rate falls for days before anyone looks, while customer feeds are wrong.

1. 1.Schema Evolution runs detect_schema_change on each delivery and compares the new layout with the last.
2. 2.assess_impact lists the fields, products and feeds the change touches, and validate_schema_compatibility checks the mapping against the downstream model.
3. 3.generate_migration drafts the mapping as a reviewed diff with a rollback point; Quality Monitoring confirms coverage and fill rate held.
4. 4.A data engineer approves every diff that renames, removes or retypes a field, and rollback_migration reverses it if verification fails. Additive changes from known suppliers are where a source can earn L3.

6. Carry licence and consent with the data, and let a person decide

A field can be loaded, joined and sold before anyone asks whether the contract allows it. For personal data the question is sharper: under the GDPR a company that obtains personal data from a source other than the individual still has notice obligations (Article 14) and needs a lawful basis for each use (regulation text), so provenance has to travel with the record.

Why it's harder than it looks: the people who read the licence never see the pipeline, and the people who build it never see the licence. An agent can compare the two, but whether a use is allowed is a legal judgement, so it stays with a named approver.

1. 1.Data Access & Governance runs scan_pii on the landing layer to flag personal data before it reaches a shared layer, and the Data Context & Catalog agent tags it with auto_tag_dataset. See how Data Workers handles PII.
2. 2.The licence terms and consent basis your team records sit next to the data, and check_policy flags a proposed use they do not clearly allow.
3. 3.request_governance_review routes the case, with the terms and the products affected, to the partnerships lead or counsel.
4. 4.The named approver decides, and generate_audit_report records who proposed the use, who approved it and when.

By task class, autonomy where work is reversible and a named approver where it is not:

7. Renew on net new coverage per dollar, not on the account manager's deck

At renewal three suppliers cover overlapping companies, places or securities, and the comparison is a one-off sample.

Why it's harder than it looks: overlap has to be measured in your product's own entity space, which needs lineage from each supplier's fields through the warehouse. Measured by hand it costs a week at the worst moment.

1. 1.The Data Context & Catalog agent traces each product field to the suppliers behind it with trace_cross_platform_lineage.
2. 2.Quality Monitoring scores each supplier on completeness, freshness and stability with get_quality_score, continuously rather than at renewal.
3. 3.The Cost Savings & Data Cleanup agent finds sources no product reads with detect_dead_assets, and get_cost_enrichment adds cost per source.
4. 4.The sourcing lead opens each renewal with net new coverage and cost per net new record per supplier, plus a list of feeds to drop. Negotiation and the decision stay with the contract owner.

Designing the autonomy ladder

Autonomy is set per task class and per source. L0 is off, L1 observes, L2 proposes and a person approves, L3 acts reversibly with a rollback point, L4 is fully autonomous. Every source starts at L1. L0 to L2 run today. L3 and L4 are where a domain can go on the evidence of its own review record ("On evidence" below), never a level reached by a date.

The operating model: who owns what

Agents change who does the work, not who is accountable for it:

The swarm behind the practices

No single agent delivers these practices; each one is a hand-off between specialists under the autonomy you set per source.

Data Context Wizard holds one governed definition of each coverage and freshness measure you sell, so a scorecard, a sales deck and a customer data sheet quote the same number. Autonomous Data-Conductor verifies outcomes before a build is delivered or a source is promoted. Spellbook Data Catalog gives each supplier a plain-English page: what it covers, what you may do with it, who owns it.

What disappears

• •The customer who finds a coverage hole before you do.
• •The hand-built loader for one supplier that nobody dares touch.
• •The silent null column after a layout change.
• •The renewal argued from a sales deck.
• •The email search for what the licence says about a field.

What replaces them is quieter: every source arrives with checks, a page and a record of what it may be used for, late and thin files are held out of customer feeds with the evidence attached, and every action leaves a signed receipt you can roll back.

A Head of Data Acquisition's scorecard

Targets to bring to each quarterly review. They are targets to set, not results:

• •New supplier to first clean load: target about a day, from about two weeks today.
• •Layout change to verified mapping: target under an hour, from about three days.
• •Delivery incidents reaching a customer feed: target zero, with each held build and coverage note on record.
• •Net new coverage per licensing dollar, per supplier: measured continuously, every renewal opened on it.
• •Freshness against customer promise: supplier delivery to live product, versus the SLA in your contracts.
• •Live sources with licence terms and consent basis recorded: target every source.
• •Upkeep closed autonomously, and its token cost: targeting 88% for catalog and docs upkeep as the deployment matures.

Where this goes next

Entity resolution picks up where a supplier record meets your entity, and data delivery picks up where a source reaches a customer feed. The family sits on the data businesses page. If you source data for training rather than resale, see the report on data acquisition for foundation model teams.

The climb: from assisted to autonomous to agentic

Our thesis is that the data platform climbs the way analytics did, in stages, and that the hard part was never knowing what to do but doing it. For sourcing teams:

• •Assisted. Agents read every supplier feed and propose each fix with a receipt; licence, consent and supplier decisions go to a named approver.
• •Autonomous, under your governance. Routine work runs on its own: deliveries checked on arrival, additive layout changes mapped the day they happen, documentation kept current.
• •The agentic data business. A sourcing team that finds the next valuable dataset. People set direction, and the platform carries it out.

The pilot: how data acquisition and sourcing teams start

The pilot proves the value on your own data. Building blocks: source onboarding and the autonomy ladder.

• •What we connect. Your supplier drops, Snowflake or Databricks, dbt, Airflow or Dagster and the delivery layer, over MCP, inside your infrastructure. Nothing migrates.
• •Scope. Onboarding two new suppliers, plus layout-change handling on the feeds you already run, at L2.
• •First week. Agents run at L1 on the scoped feeds: they profile every delivery, map lineage into your products and show the fixes they would have made, before anything changes.
• •By week two. Delivery checks and documentation upkeep run on reversible work. Layout changes and anything touching rights or personal data stay at L2 with a named approver.
• •At day 30. We review against your baseline:

• •New supplier to first clean load: from about two weeks to about a day
• •Supplier layout change mapped: from about three days to under an hour
• •Overlap report across two suppliers: from about a week to ~30 min
• •catalog and docs upkeep closed autonomously, targeting 88% as the deployment matures

• •What you decide. Whether to expand to the next set of suppliers.
• •Guarantee. In a 90-day pilot, ten engineering hours per data engineer per week recovered against a baseline both sides sign on day one, or the $7,500 fee is refunded.
• •Cost. Start with a pilot ($7,500 one-time, a 6 to 12 week pilot, credited in full against the first year). Scale is from $1,000 a month and Enterprise from $3,000 a month, billed annually, with unlimited seats, no usage meter and no markup on model spend. See pricing.

By month 12, more than 42% of agent actions run fully autonomously and manual work falls below 10%, while rights, consent and supplier decisions stay with a named approver.