Adoption Dashboards by Hand vs With an AI Agent: What Changes and What Doesn't

Building an adoption dashboard by hand means an analyst defines the feature, writes the SQL, builds the chart, and then re-explains the numbers every time someone asks why they moved. An AI agent can take over the repeatable parts: building the dashboard against defined targets and interpreting the result. A human still owns the definitions and the sign-off. Our Insights Agent ships a skill for this job, dw-adoption-dashboards, described as building and interpreting adoption dashboards that track feature uptake against defined targets. This post is our opinion on where the manual workflow breaks and how to decide whether an agent is worth trying.

Key Takeaways

• •The slow part of an adoption dashboard is not the chart. It is agreeing on what counts as adopted, then keeping that definition consistent over time.
• •An agent is useful when the definition lives in a semantic layer it can read, so every number traces back to a stated metric and stated assumptions.
• •The agent builds and interprets. A human still approves the metric definition, the target, and anything that changes what leadership sees.
• •Evaluate an agent on a question you already know the answer to, not on a demo dataset.
• •Effort differences are real but vary by team. We do not quote a time saving here because we have no attributed benchmark for this specific job.

Adoption dashboards by hand vs with an AI agent: the job and who does it today

An adoption dashboard answers one question for product and go-to-market teams: is the feature we shipped actually being used, and is that use on track against the target we set? In most companies the work lands on whichever analytics engineer or analyst sits closest to the product team. They are rarely the person who decided what adoption means.

That mismatch is the root of most of the pain. The product manager knows the intent of the feature. The analyst knows the tables. The dashboard is the place where those two kinds of knowledge have to meet, and today the analyst is the only integration layer between them.

The manual workflow, step by step

We see the same sequence in most teams. Each step has a tool and a handoff, and each handoff is a place where meaning leaks.

• •Define adoption. A product manager writes a sentence such as 'accounts that used the feature in the last 28 days'. The sentence is ambiguous about accounts versus users, about what counts as use, and about the window.
• •Find the events or tables. The analyst searches the warehouse for the right event table, often asking a colleague which of several similarly named tables is current.
• •Write the SQL. The analyst joins usage to accounts, applies the window, and handles edge cases such as internal users, trial accounts, and deleted records.
• •Build the visual layer. The query is wired into a BI tool, with a target line, filters and a date range.
• •Set the target. Someone supplies a number. It is often written into the chart itself rather than stored as a governed value.
• •Interpret and report. When uptake dips or stalls, the analyst investigates cohorts, segments and releases, then writes the explanation in a message or a slide.
• •Maintain it. The definition drifts, a source table is renamed, and the dashboard quietly reports something different from what the original request meant.

Where the manual workflow breaks

Three failure modes show up repeatedly. First, the definition of adoption is stored in someone's head or in a chart title, so two dashboards built by two people disagree. Second, interpretation is a separate piece of work from construction, so the dashboard shows that adoption fell but not why, and the follow-up question becomes a new ticket. Third, maintenance is invisible until a number looks wrong in a meeting.

None of these are tooling failures in the narrow sense. They are coordination failures between the people who define a metric and the people who compute it. That is the part an agent can help with, and also the part where an agent can do damage if it guesses.

What an agent does instead

We want to stay inside what the skill actually covers. The dw-adoption-dashboards skill is described as building and interpreting adoption dashboards that track feature uptake against defined targets. We read that as two jobs: construct the dashboard from defined metrics and targets, and explain the result.

The Insights Agent is text-to-SQL grounded by the semantic layer, and it produces analyses with citations and assumptions. In practice that means a question like 'how is adoption of the new export feature tracking against target?' is answered from a stated metric definition rather than from a table the model picked by guesswork. The answer lists the assumptions it relied on, so a reviewer can challenge them.

Because every Data Workers agent is an MCP server, the request can come from the place an engineer already works. The Model Context Protocol is the open standard that lets a client such as Claude Code call tools exposed by a server. That makes the dashboard job a terminal command instead of a ticket in another queue.

The agent also chains with neighbours. Our Catalog Agent) supplies what the tables and columns mean, and our Quality Agent can say whether the underlying data is fresh enough to trust. We covered how we keep that context current in a separate post on how Data Workers keeps the catalog current.

What still needs a human approval

An agent that builds and interprets is not an agent that decides. We keep these with a person:

• •The definition of adoption. If the metric is wrong, a faster dashboard is only a faster way to report the wrong thing.
• •The target. A target is a business commitment, not a computed value.
• •Publishing to leadership or customers. Anything that changes what an executive or an outside party sees should have a named approver.
• •Changes to shared metrics. Editing a governed definition affects every other dashboard that uses it.

This lines up with how the open-source edition behaves. The OSS edition, dw-claw, is Apache 2.0 and read-only by default, so an agent can analyse without being able to change anything. Pro adds writes along with approval workflows and audit trails for teams that want the agent to publish.

Manual vs agent, side by side

How to evaluate an agent for this job

We would not judge an agent on a polished demo. Run it on a feature whose adoption number your team already trusts, and check these things:

• •Does it state its assumptions? An answer without the window, the account definition and the exclusions is not reviewable.
• •Does it use your metric definition or invent one? Ask the same question twice and compare.
• •Can it say why a number moved, not only that it moved? Test it against a known release or a known data issue.
• •Does it refuse or flag when the data is stale or the definition is ambiguous?
• •What can it write? Start with read-only access and add write permission only when the reads have earned trust.
• •Is there an audit trail? You should be able to see what was asked, what ran, and who approved what.

If you are running Claude Code or Cursor today, the cheapest test is to connect the agent as an MCP server, point it at a non-production copy of the data, and ask the five or six questions your product team asks every month.

Frequently Asked Questions

Can an AI agent build an adoption dashboard on its own? It can build one from a defined metric and target, and it can interpret the result. It should not invent the definition of adoption or the target. Those are decisions a person owns, and we keep them behind approval.

How much time does an agent save compared with building the dashboard by hand? It depends on how well your metrics are defined. We do not have an attributed benchmark for this specific job, so we describe the saving qualitatively: the agent removes most of the query writing and the follow-up interpretation requests, and leaves definition and sign-off with your team.

Does the agent need write access to the warehouse? No. The open-source edition is read-only by default, which is enough to analyse adoption and explain it. Write access and approval workflows are part of Pro.

Where does the data go when the agent runs? Data Workers is built so that data never leaves the customer's infrastructure. The agent runs against your environment and the framework applies access control, PII redaction and audit logging across every agent.

Which tools can I call the Insights Agent from? Any MCP-compatible client, including Claude Code, Cursor, Windsurf, VS Code and GitHub Copilot, because each agent is an MCP server.