Onboard new data sources with agents, documented and tested
Every new data source starts as a ticket, then weeks of access requests, profiling, documentation and tests before anyone trusts it. Data Workers connects the source read-only, profiles it, documents it and writes its tests, with a named approver before anything goes live.
Before and after
| Today | With Data Workers |
|---|---|
| A ticket per source, waiting in the platform queue | Sources connected read-only, with each connection tested |
| Missing grants found one failed attempt at a time | The agent lists exactly what access is missing |
| A new table nobody has profiled until it breaks | Every new table profiled before it reaches a dashboard |
| Documentation written after the fact, if at all | Table and column documentation drafted on arrival |
| Tests added last, or skipped | Tests written with the source and run before it goes live |
Every business team wants another source in the warehouse: a CRM, a billing system, a partner feed, a product event stream. The connector is rarely the slow part. The slow part is the access requests, the first look at the data, the documentation and the tests, and that queue grows with every source added. Moving a feed is covered in Inside the Connectors Agent; this page is about everything that has to happen after it connects.
How it works
1. Connect and check
You give the agents read-only credentials. The Connectors Agent (get_connector_status) tests the connection and tells you exactly what is missing, so access gaps surface in the first hour, not after three failed attempts. It works across 50+ connectors, including warehouses, orchestrators, streaming and ingestion tools such as Fivetran and Airbyte. Nothing is copied out of your stack.
2. Profile and document
The Quality Monitoring Agent (run_quality_check) profiles each new table and starts learning its normal behaviour. The Data Context & Catalog Agent (generate_documentation) drafts what each table is, how it is built, what it feeds and who owns it, and flags what it could not work out (flag_documentation_gap). The Data Ingestion Agent handles the landing side. If you are weighing the two ways of bringing data in, see ingestion vs integration.
3. Test and hand over
The Quality Monitoring Agent (create_quality_tests_for_pipeline) writes the tests that belong with the new source, and the Pipeline Building Agent (validate_pipeline) runs them before anything goes live. A named approver signs off, and the source is registered with its owner and lineage. Every step lands in the audit trail.
The impact metrics we move
- •Connection checked, access gaps listed: from about 1 day to 1 h
- •New tables profiled and documented: from about 3 days to 3 h
- •Tests written for a new source: from about 1 day to under 1 h
Where teams start
- •SaaS sources behind the intake queue. The CRM, billing and support exports that wait longest for a data engineer.
- •Warehouse-to-warehouse additions. A new Snowflake, BigQuery or Databricks source after an acquisition or a team reorg.
- •Fivetran or Airbyte syncs. New connectors landed in the warehouse with documentation and tests the sync never provided.
- •Event streams. A new Kafka topic registered, profiled and owned before the first dashboard reads it.
- •Partner and vendor feeds. Files and APIs from outside, checked on arrival against what the contract promised.
What powers it
The Connectors Agent tests each connection, the Data Ingestion Agent handles landing, the Quality Monitoring Agent profiles and writes tests, and the Data Context & Catalog Agent documents what arrives. The Data Context Wizard supplies what each table means and which source is trusted, and the Spellbook Data Catalog (in preview) carries the page for each new source. Autopilot Standards keep every onboarding to your conventions, and the Autonomous Data-Conductor coordinates the agents and keeps what waits for approval in one view. This shows up in data acquisition and sourcing and in instrumentation QA, and it pairs with agent-built pipelines. For a platform team's view of the same queue, read stop onboarding sources by ticket.
The climb: from assisted to autonomous to agentic
Agents start by watching and suggesting: checking connections, profiling tables and drafting documentation for people to review. On evidence, a domain can move to agents onboarding routine sources inside approved standards, with a named approver on anything sensitive. L0 to L2 run today; L3 and L4 are where a domain can go as the track record builds. See our thesis.
The pilot
- •Scope. Three new sources from your intake queue, connected read-only.
- •First week. Agents start by watching and suggesting: they check each connection, profile the tables and draft documentation for your engineers to review. Nothing goes live.
- •By week two. Routine sources go through the full path, from connected to profiled to documented to tested. Anything sensitive stays with a named approver.
- •At day 30. We review the impact metrics against your baseline:
- •Connection checked, access gaps listed: from about 1 day to 1 h
- •New tables profiled and documented: from about 3 days to 3 h
- •Tests written for a new source: from about 1 day to under 1 h
- •Cost. Start with a pilot ($7,500 one-time, credited in full against the first year). Scale is from $1,000 a month and Enterprise from $3,000 a month, billed annually, with unlimited seats, no usage meter and no markup on model spend. See pricing.
FAQ
Does it need write access to our sources? No. You start with read-only credentials, and the agents tell you exactly what access is missing.
Do we have to move or copy our data? No. The agents connect to the warehouse, catalog and orchestrator you already run.
Which stacks does it cover? 50+ connectors across warehouses, orchestrators, streaming, ingestion, catalogs, quality and BI tools. Check your specific source against the list during the pilot; a source without a connector stays with your engineers, and the Pipeline Building Agent drafts the pipeline for their review.
What if the documentation is wrong? It is a draft until a named approver accepts it, and owners can correct any page. The agents flag what they could not work out instead of guessing.
What stays with our engineers? Deciding which sources matter, the access grants and the sign-off. The agents take the checking, profiling, documentation and tests.