Governed product metrics, run by agents
DAU, activation and retention each have a few defensible definitions, and the dashboard, the notebook and the AI assistant each pick a different one. Here is how Data Workers holds one definition of each across all three, with every change approved by the product analytics lead.
Key takeaways
- •The problem today. DAU, activation and retention are defined in the BI tool, in the semantic layer, in analysts' notebooks and in whatever an AI assistant decides. Reviews open with an argument about which number is right.
- •What changes. Definitions live in one governed semantic layer. Agents answer from it, flag any query that computes a metric differently, and show what a proposed change would move before anyone approves it.
- •What stays with your people. Every change to a definition of DAU, activation, retention or the north star metric goes to the product analytics lead. Agents propose; they never publish.
- •The result. BI, notebooks and AI agents report the same number, and a change to a definition has a named approver and a record.
The problem today
A PM pastes "DAU: 412k" into the Monday review. The data scientist's notebook says 388k. The board deck, built last quarter and patched since, says 401k. Nobody made an arithmetic error.
The dashboard counts any event, the notebook counts users with a qualifying action, and the deck uses account time zones where the others use UTC. Activation is "created a project" in one place and "invited a teammate within seven days" in another. Retention is day-N for growth and a rolling window for finance. Then an AI assistant writes SQL against the first table that looks right.
Each definition is defensible. Together they leave the product and usage data teams proving the number before they can discuss it. Airbnb describes the same problem at scale in its write-up on metric consistency. The owners sit in the clickstream and product analytics and usage and billing data playbooks.
What changes with Data Workers
Agents treat a metric definition as a governed asset with an owner, not a query somebody wrote. Data Context Wizard holds one approved definition each of DAU, activation, retention and the north star. It reads the ones you already keep, such as dbt MetricFlow or Cube, AtScale and others, so nothing is rewritten into a new layer. If your charts live in Amplitude, Data Workers checks the warehouse tables behind them. Analysts, notebooks and AI agents reach the same definitions through the same governed route, and the agent names the one it used. This is the product-metrics case of the general semantic layer pattern.
The workflow, step by step
- 1.The Data Context & Catalog agent calls
list_semantic_definitionsand lists every definition of DAU, activation and retention in use across the semantic layer, the BI tool and the notebooks you connect, and where they diverge. - 2.The Data Context & Catalog agent calls
resolve_metricwhenever a dashboard query, an analyst or an AI agent asks for one of these metrics. It answers from the approved definition and says so. When two definitions disagree, it flags the conflict and points to the authoritative one. - 3.The Data Context & Catalog agent calls
blast_radius_analysiswhen someone proposes a change, such as excluding internal accounts from DAU. It lists the dashboards, readouts and models whose numbers would move. - 4.The Data Access & Governance agent calls
request_governance_reviewand sends the change, with that list and a way to undo it, to the product analytics lead. The Autonomous Data-Conductor coordinates the sequence and keeps each step inside the autonomy your owners allow.
Background: business definitions for AI agents. Upstream, instrumentation QA keeps the events these metrics count trustworthy, and experiment readouts read the same activation definition.
The impact metrics we move
- •Every active-user definition found and compared: from about 2 days to about 30 min
- •Board deck vs. product dashboard DAU tie-out: from about 1 day to 15 min
- •Impact of a proposed definition change: from about 2 days to about 30 min
- •"Which definition is this?" answered: from about 4 h to seconds
- •Definition change ready for the owner to approve: from about 2 days to about 2 h
What goes on autopilot
- •Drift checks. Dashboards and saved queries that report DAU, activation or retention are compared with the approved definition every day, so a quietly edited filter shows up as a flagged difference, not a surprise in the review.
- •Consistent answers. A PM's question in Slack, an analyst's notebook and a coding agent resolve to the same definition and cite it.
Where your people stay in charge
The product analytics lead approves every change to a definition of DAU, activation, retention or the north star. Which events count as "active" is a product judgement, and moving it restates history, so agents propose and the lead decides. Restating a period already shown to the board goes to the same approver, and choosing the north star stays with product leadership. Each approved change carries a signed record of who approved what and when, so "when did DAU change?" has an answer. Everything starts at L1, observing only, and autonomy rises only on evidence.
The pilot
- •Scope. DAU, activation and weekly retention for one product area, at L2: agents flag and propose, the product analytics lead approves.
- •First week. Agents run at L1, observing only, and show the conflicting definitions and drifting dashboards they found, with receipts.
- •By week two. Daily drift checks and definition-aware answers run on their own; every definition change stays with the product analytics lead.
- •At day 30. We review the impact metrics above against your baseline.
- •Cost. Start with a pilot ($7,500 one-time, credited in full against the first year). Scale is from $1,000 a month and Enterprise from $3,000 a month, billed annually, with unlimited seats, no usage meter and no markup on model spend. See pricing.