Engineering
Engineering13 min readBy The Data Workers Team

Best practices for agentic financial crime and AML data: a field guide for financial crime data teams

Transaction monitoring, KYC and sanctions screening are only as good as the data under them, and the data changes every week without telling anyone. This field guide sets out how to keep AML data complete, current and traceable with agents, practice by practice, with the old world, the new world, who approves what, and the numbers a Head of Financial Crime Data should track. Investigators and officers still make every decision on customers, alerts and filings.

Ask the head of financial crime data what keeps the team up and it is rarely the model. It is the field that went empty after a core banking release, the new payment rail nobody mapped, the sanctions file whose layout changed on a Friday. A scenario that depends on counterparty country does nothing for transactions where that field is empty, and it does not say so. The gap shows up in a validation, an audit or an exam, long after the transactions went through unmonitored.

The mandate is plain: keep the monitored population complete, keep customer and list data current, and show, for any day and any alert, what was in scope and where each field came from. The FATF Recommendations frame the expectation as customer due diligence, ongoing monitoring, reconstructable records and suspicious activity reporting. Each rests on data that has to be there.

Our view, as product people who have run data platforms: the work that keeps AML data complete is repeatable, and agents should own it. The work that decides whether activity is suspicious stays with named investigators and officers. This report is the field guide to that split.

The old world

It is 07:30 on a Monday. A payments hub released on Saturday night with a new instant-payment channel code, and counterparty country moved to a renamed field. Both loads succeeded. Four scenarios that read those fields now skip the new rail, and the batch runs at 08:00 on a population nobody has checked. The sanctions provider changed a file layout last week, and nobody can say with confidence which version screened Friday's payments. A validator has asked for the lineage behind one alert, and an analyst has opened a spreadsheet. The channel problem surfaces in three weeks, when a validation sample comes up short.

The new world

The same Monday with Data Workers. The Quality Monitoring agent tied source to the monitored population overnight and found new-channel transactions belonging to no monitored product. The Schema Evolution agent traced it to Saturday's release, the Data Context & Catalog agent listed the four scenarios and one report input that depend on the fields, and the Pipeline Building agent drafted the mapping with tests. The Autonomous Data-Conductor held it for the owner of the monitoring inputs, who approved before the batch. The list update had been checked on arrival and recorded with its version, and the validator's request is answered from live lineage.

Here is that new world as a chain of hand-offs between agents:

How the swarm hands one problem from agent to agent: a release that dropped a payment channel out of scope. Each step leaves a signed receipt.
Fig. 1:How the swarm hands one problem from agent to agent: a release that dropped a payment channel out of scope. Each step leaves a signed receipt.

The full playbook walks through this scenario and two more in depth. This report covers the 7 practices behind it. At a glance:

The practices in this report, old world vs. new world. Each row is a practice; the right column is how it runs when agents own the data back office. Illustrative.
Fig. 2:The practices in this report, old world vs. new world. Each row is a practice; the right column is how it runs when agents own the data back office. Illustrative.

The practices, one by one

1. Tie every source feed to the monitored population every morning

If a product, channel or currency is missing from the monitored population, no scenario can be right about that slice. A completeness check run once a year for an audit finds problems a year late.

Why it's harder than it looks: Source and monitoring differ for boring reasons: a product filter, a currency code that no longer maps, a late file. The right difference is not zero, so you must know which gaps are expected, and that list changes whenever a source does.

  1. 1.The Connectors agent confirms each feed landed with get_connector_status.
  2. 2.The Quality Monitoring agent compares source transactions with the monitored population by product, channel and currency with run_quality_check, against an SLA set with set_sla.
  3. 3.get_anomalies separates a documented exclusion from a new one, and the Incident Debugging agent names the change behind it with get_root_cause.
  4. 4.The monitoring input owner approves any change to what counts as in scope, with the diff in front of them.

What to measure. Days on which every feed ties out, and time from a new gap to its explanation.

2. Check rule-critical fields on arrival, not at validation

Every scenario has fields it cannot do without. When one goes null or is renamed, the rule silently covers less. The check belongs where the data lands.

Why it's harder than it looks: A field is critical only to some rules, and nobody has the full list written down. Checks cover the fields someone remembered, and the quiet failure happens on the one nobody did.

  1. 1.The Data Context & Catalog agent lists which rules, reports and profiles read each field, with blast_radius_analysis.
  2. 2.The Quality Monitoring agent generates a test per critical field with create_quality_tests_for_pipeline, so coverage follows the dependency list, not memory.
  3. 3.On each load, run_quality_check flags an empty or out-of-range field with the transactions affected.
  4. 4.The Pipeline Building agent proposes the fix and tests it with validate_pipeline in staging. The monitoring input owner approves it.

What to measure. Silent exclusions found by validation or audit rather than by the data team. The target is zero.

Time to complete 6 routine financial crime data tasks, current practice (black) vs. the agent swarm (green). Illustrative.
Fig. 3:Time to complete 6 routine financial crime data tasks, current practice (black) vs. the agent swarm (green). Illustrative.

The biggest change is the model validation data pack, from about 2 weeks to about a day. Work that waited in queues finishes in minutes or hours, because agents detect, trace and prepare the evidence in one pass.

3. Map source changes before the next monitoring run

New rails, releases and acquired books arrive with new codes and renamed fields. The mapping should exist before the next batch, not after the first quiet week.

Why it's harder than it looks: Changes are announced to the platform team, not the data team. The ones that matter are rarely new columns. They are a changed meaning, such as a code reused for another channel, which no schema comparison flags.

  1. 1.The Schema Evolution agent runs detect_schema_change on each arrival and check_compatibility against the contract monitoring depends on. See schema changes that never break downstream.
  2. 2.The Data Context & Catalog agent runs assess_impact to show every rule input affected.
  3. 3.The Pipeline Building agent drafts the mapping with generate_pipeline. The Autonomous Data-Conductor holds it and sends one decision with the diff, the affected scenarios and the way back via rollback_migration.
  4. 4.The monitoring input owner approves. Any scenario or threshold change the new data suggests goes to its owner, not to an agent.

What to measure. Time from a source change to a verified, approved mapping, against time to the next monitoring run.

4. Land and version sanctions and PEP lists as they arrive

Screening is only as current as the list it ran against, and you need to know which version was in force. OFAC publishes a machine-readable Sanctions List Service.

Why it's harder than it looks: A load that succeeds is not a load that is right. Malformed dates, lost aliases and truncated files all load, and proving the version later means digging through logs.

  1. 1.The Connectors agent reports each list arrival with get_connector_status.
  2. 2.The Quality Monitoring agent checks completeness and format against the previous version with run_quality_check.
  3. 3.The Data Context & Catalog agent tracks freshness against the SLA with check_freshness, and the Data Access & Governance agent records the version loaded with generate_audit_report.
  4. 4.The sanctions screening data lead approves any change to how a list is loaded or mapped. Match handling stays with the sanctions team.

What to measure. List updates verified and versioned within SLA, and time to show the version in force at a given moment.

5. Watch entity resolution output instead of replacing the engine

Your matching engine resolves identities. The practice is to watch what it produces, because risk ratings computed on the wrong record are wrong in ways nobody sees.

Why it's harder than it looks: Resolution quality drifts quietly. A new source changes name formats and the match rate slides a point. Nobody is paged for a point.

  1. 1.The Quality Monitoring agent surfaces orphaned identifiers, unexpected duplicates and match-rate shifts daily with get_anomalies.
  2. 2.The Data Context & Catalog agent traces each case to its source record with trace_cross_platform_lineage.
  3. 3.The Incident Debugging agent checks get_incident_history for whether this pattern has happened before.
  4. 4.The KYC data product owner gets a short list with evidence and decides whether to fix a source or raise it with the engine's owner.

What to measure. Orphaned and duplicate identifiers open longer than a day.

6. Generate lineage and evidence from live lineage

Examiners, validators and audit ask one question: where did this come from, what did the rule see, and has it moved? If the answer is a project, the team is a bottleneck.

Why it's harder than it looks: Lineage built by hand is stale the day it is written, model input drift lives in a second place and approvals in a third.

  1. 1.The Data Context & Catalog agent traces source record to alert with trace_cross_platform_lineage; see end-to-end lineage.
  2. 2.The Observability agent shows what each agent did with get_audit_trail, and the MLOps & Models agent shows input movement with detect_model_drift and get_model_versions.
  3. 3.The Data Access & Governance agent packages change history, owners and approvals with generate_audit_report; see audit evidence on demand.
  4. 4.The model validation data analyst reviews the pack. The validator's conclusions stay the validator's.

What to measure. Time from an evidence request on one alert to a complete pack.

7. Keep decisions with named people, and scope access to case and SAR data

Case and SAR data is the most restricted data you hold, and speed in the data layer must not bleed into decisions about customers.

Why it's harder than it looks: The failure is rarely dramatic: an access grant nobody remembers, or a helpful summary that quietly becomes the decision. See zero-ticket data access.

  1. 1.The Data Access & Governance agent applies check_policy and enforce_rbac so agents act under scoped identities, and case and SAR tables can be excluded entirely. scan_pii classifies customer attributes.
  2. 2.request_governance_review sends any access change to the data owner, and every grant lands in the audit trail.
  3. 3.Alert disposition, case closure, risk-rating overrides, exits and SAR or STR filing stay with named investigators and officers. Agents assemble the source data they ask for. They do not decide, draft the decision or file.
  4. 4.Scenario, threshold and model changes go to their owners and your model risk process.

What to measure. Share of access to case and SAR data with a named approver and a logged grant. The target is all of it.

That split, autonomy where work is reversible and a named person where it matters, looks like this by task class:

Autonomous resolution by financial crime data task class. Share of each task class closed autonomously, escalated for approval, or handled by a person. Decisions on alerts, cases and filings are never automated. Illustrative.
Fig. 4:Autonomous resolution by financial crime data task class. Share of each task class closed autonomously, escalated for approval, or handled by a person. Decisions on alerts, cases and filings are never automated. Illustrative.

Lineage and docs upkeep closes autonomously 86% of the time in a mature deployment. For alert, case and filing decisions, autonomous closure is 0% by design.

Designing the autonomy ladder

Autonomy is set per task class, not per tool. L0 is watch only, L1 observe and report, L2 propose and wait for a named approver; those run today. L3 (act with a rollback point) and L4 (act on its own) are where a domain can go on evidence from its own review record. Here is how we would set this team's classes:

Task classRuns today (L0 to L2)Can go to on evidence (L3, L4)
Tie-outs and field checksL1 to L2: daily checks reported to a named personL3 for routine checks
Lineage and documentation upkeepL2: owners approve updatesL3, then L4 for descriptions and tags
Sanctions and PEP list verificationL1 to L2: loads changes held for the sanctions leadL3 for format and completeness checks
Mapping changes to monitoring inputsL2: full diff to the input ownerStays at L2 until the owner raises it
Access to KYC, case and SAR dataL2: data owner reviewsStays at L2
Alert, case and filing decisions; scenarios, thresholds, modelsL0 for agentsStays at L0 by design

Only a domain's owner can raise its level. If people start reverting an agent's changes, that domain pauses and hands back to a person.

The operating model: who owns what

Agents change who does the work, not who is accountable for it:

RoleOwnsWhat the agents take off their plate
Head of Financial Crime DataStandards, autonomy per domain, which domain goes nextAssembling status and rebuilding the same completeness proof
Monitoring input ownerMapping changes and the definition of in scopePreparing the diff, blast radius and rollback plan
Sanctions screening data leadList loading, versioning and freshness SLAsChecking each update and recording the version
KYC data product ownerCustomer and counterparty data qualitySurfacing orphans, duplicates and match-rate shifts
Model validation data analystEvidence for the validatorAssembling inputs, lineage, drift and history
BSA or AML officer and investigatorsEvery decision on customers, alerts, cases and filingsPulling together the source data they ask for

The swarm behind the practices

No single agent delivers these practices. Each is a hand-off between specialists, coordinated under the autonomy you set per domain.

AgentWhat it doesPublic tools
ConnectorsLand core banking, payments, KYC and list feedsget_connector_status
Schema EvolutionDetect source changes against the contracts monitoring needsdetect_schema_change, check_compatibility, rollback_migration
Quality MonitoringDaily tie-outs, field checks and list verificationrun_quality_check, set_sla, get_anomalies
Incident DebuggingTrace a gap to the change behind itget_root_cause, get_incident_history
Data Context & CatalogLineage, blast radius, freshnessblast_radius_analysis, trace_cross_platform_lineage, check_freshness
Data Access & GovernanceScoped access and evidence packscheck_policy, enforce_rbac, generate_audit_report
ObservabilityThe audit trail of agent actionsget_audit_trail
MLOps & ModelsInput drift and version historydetect_model_drift, get_model_versions
Pipeline BuildingBuild and validate monitoring inputsgenerate_pipeline, validate_pipeline

Data Context Wizard holds one governed definition of each rule input, customer attribute, list and monitored product, so the monitoring platform, validators and investigators read the same meaning. Autonomous Data-Conductor verifies the numbers that matter before the batch runs. Spellbook Data Catalog makes each data set discoverable with owner, lineage and approvals. They sit beside your monitoring, screening and case platforms. For the neighbouring problem of fraud feature freshness, see fraud detection data, run by agents.

What disappears

The clearest sign the practices are working is what stops happening:

  • •The quiet week on a rule that should have fired.
  • •A channel or currency running outside monitoring for weeks.
  • •The Friday sanctions file nobody can vouch for.
  • •The spreadsheet that traces one alert back to source.
  • •The two-week scramble before a model validation or exam.

Gaps are caught where data enters, mappings arrive with tests, and every action leaves a signed receipt in a tamper-evident audit trail. For governance depth, see data governance for fintech, the audit trail requirements for autonomous data agents and how approvals work for AI data agents. Our AI model governance guide covers the model side, including the April 2026 SR 26-2 guidance, which your model risk team interprets.

A Head of Financial Crime Data's scorecard

If you own this data, these are the numbers to bring to every quarterly review. They are targets to track, not results:

  • •Days on which every feed ties to the monitored population: every day, every feed.
  • •Time from a source change to a verified, approved mapping: before the next monitoring run.
  • •Silent exclusions found by validation, audit or an examiner instead of by the team: zero.
  • •Sanctions and PEP list updates verified and versioned within SLA: all of them.
  • •Time to assemble the evidence pack for one alert: from about 2 days to about 1 h.
  • •Share of lineage and docs upkeep closed autonomously, and its token cost: targeting 86% as the deployment matures.

The climb: from assisted to autonomous to agentic

Our thesis is that the data platform climbs the way analytics did, in stages, and that the hard part of data work was never knowing what to do but doing it. For financial crime and AML data teams, the climb looks like this:

  • •Assisted. Agents work alongside your team. They read the whole stack, show what needs doing and propose every fix with a receipt; mapping changes to monitoring inputs always go to a named approver.
  • •Autonomous, under your governance. Tie-outs, field checks and list verification run on their own. Your team governs the agents instead of doing the upkeep.
  • •The agentic enterprise. Monitoring data is complete every morning, lineage is current, and an exam request is a query, not a project. People set direction; investigators decide, with the data ready.

You choose the altitude, every step is visible and reversible, and the pilot below is the first rung.

The pilot: how financial crime and AML data teams start

The pilot is how your team proves the value on its own data before committing. These teams sit in the larger finance, insurance and investment data teams group, and the pilot is scoped to one slice of it.

  • •What we connect. Core banking and payments feeds; Snowflake or Databricks; Airflow or Control-M; dbt; and your monitoring and screening platforms, over MCP, inside your infrastructure. Nothing migrates.
  • •Scope. Source-to-monitoring completeness tie-out on one product line at L2.
  • •First week. Agents run at L1 across the scoped domain. They map every pipeline, definition and lineage path and show the fixes they would have made, with receipts, before anything changes.
  • •By week two. Daily source-to-monitoring tie-outs and rule-critical field checks start running on reversible work. Mapping changes to monitoring inputs stay at L2 with a named approver.
  • •At day 30. We review the impact metrics against your baseline:
  • •Source-to-monitoring tie-out: from about a week to 10-15 min
  • •Silent exclusion explained: from about a week to under 1 h
  • •Alert evidence pack: from about 2 days to about 1 h
  • •the share of lineage and docs upkeep closed autonomously, targeting 86% as the deployment matures
  • •What you decide. Whether to expand to the next product line or to sanctions list data. The pilot fee is credited in full against the first year.
  • •Cost. Start with a pilot ($7,500 one-time, a 6 to 12 week pilot, credited in full against the first year). Scale is from $1,000 a month and Enterprise from $3,000 a month, billed annually, with unlimited seats, no usage meter and no markup on model spend. See pricing.
  • •Pilot guarantee. In a 90-day pilot, ten engineering hours per data engineer per week are recovered against a baseline both sides sign on day one, or the $7,500 fee is refunded.

The pilot is the start of a longer curve:

Share of agent actions by autonomy rung over the first twelve months. Month 0 is the manual baseline; month 0 is the manual baseline, and every deployment starts at L1 and moves up on evidence. Illustrative.
Fig. 5:Share of agent actions by autonomy rung over the first twelve months. Month 0 is the manual baseline; month 0 is the manual baseline, and every deployment starts at L1 and moves up on evidence. Illustrative.

By month 12, more than 36% of agent actions run fully autonomously while decisions on alerts, cases and filings stay with named people.