Orchestrating Insights, Catalog and Observability Agents Together

Orchestrating an Insights agent, a Catalog agent and an Observability agent means giving each one a single verb: Observability detects, Catalog decides what the data means, and Insights acts by producing the answer. Every handoff passes a typed context object over MCP, and nothing writes to the warehouse without a human approval. This post lays out that framework, the handoff table, and the failure modes we plan for.

Key Takeaways

• •Split the job by verb: the Observability Agent detects, the Catalog Agent decides what a table or metric means, and the Insights Agent acts by answering.
• •An Insights answer should never ship without a freshness verdict from Observability and a semantic definition from Catalog attached to it.
• •Each handoff carries a small, typed context object (asset IDs, freshness verdict, definition, lineage path), not a free-text summary.
• •Approval points belong where an agent would write, publish or notify at scale; read-only analysis can run unattended.
• •When one agent fails, the other two degrade to a labelled partial answer instead of guessing.

The shared job: answer a data question you can trust

The three agents share one job: take a question like "Why did row counts drop 40% overnight?" and return an answer that states what the data says, whether the data is fresh, and what the numbers mean. A text-to-SQL agent alone can write the query. It cannot know that the upstream table is six hours stale, or that "active customer" was redefined last quarter. Those facts live in other systems.

That is the case for splitting responsibility. The Insights Agent writes SQL grounded by the semantic layer and returns analyses with citations and assumptions. The Catalog Agent federates metadata across OpenMetadata, DataHub, Amundsen, Unity Catalog and Atlan, so definitions come from wherever your team already keeps them. The Observability Agent owns freshness SLAs, lineage traversal and log, metric and trace correlation.

Warehouse vendors are converging on the same shape from the other direction. Databricks describes Genie One as a chat interface that first looks for a matching Genie Agent, then searches dashboards, queries and metric views, and its docs note that workspaces with many Genie Agents may see reduced routing accuracy, per the Genie One documentation. Routing quality becomes a design problem as soon as more than one agent is involved, which is why we make the routing explicit.

Who detects, who decides, who acts

We assign roles by what each agent is allowed to conclude. The Observability Agent detects: it can say a table is late, a volume shifted, or an upstream job failed. It cannot say what a metric means. The Catalog Agent) decides: it resolves which table is the governed source, what the column definitions are, and who owns the asset. It cannot say whether the data arrived on time.

The Insights Agent acts, but its action is narrow. It composes the query and the narrative, then attaches the verdicts from the other two. If either verdict is missing, the answer is labelled partial. This keeps the final response honest, because the agent that talks to the user is the only one that never gets to hide the other agents' uncertainty.

In larger setups an Orchestration Agent sits above the three and delegates. It does not add knowledge. It owns ordering, timeouts and retries, so the three specialists stay simple. For a three-agent setup you can run that logic in a short script instead; the handoff contract matters more than who executes it.

The five-step flow

• •Step 1, intake: the question arrives from Claude Code, Cursor or any MCP client, and the orchestrator extracts candidate entities (tables, metrics, time range).
• •Step 2, resolve: the Catalog Agent maps those entities to governed assets and returns IDs, definitions, owners and certification status.
• •Step 3, verify: the Observability Agent checks freshness and recent incidents for exactly those asset IDs and traverses upstream lineage one or two hops.
• •Step 4, answer: the Insights Agent runs read-only SQL against the resolved assets and writes the analysis with citations and stated assumptions.
• •Step 5, review: the orchestrator checks that every claim carries a source and a freshness verdict, then returns the answer or escalates to a human.

Notice the order. Catalog goes before Observability because observability checks are only meaningful once you know which assets are the governed ones. Checking freshness on a deprecated copy of the orders table produces a clean verdict about the wrong table.

The handoff table

A handoff is a contract: what the sender guarantees, and what the receiver is allowed to assume. We keep the payloads small and structured so a receiving agent never has to parse prose.

The "timestamp" field matters most. A freshness verdict that is an hour old is not the same as one from ten seconds ago, and the Insights Agent should say which one it used.

Approval points

Our open-source edition is read-only by default, which covers everything in the flow above. Approval becomes necessary the moment an agent would change something: pausing a pipeline, editing a certification tag, posting an answer to a shared channel, or opening a ticket on someone else's behalf.

This matches the stance in the MCP specification, which says hosts must obtain explicit user consent before invoking any tool and that tool descriptions should be treated as untrusted unless they come from a trusted server, per the MCP security principles. We apply it at the workflow level too: one approval covers a proposed action with its evidence bundle, instead of a prompt on every tool call.

In practice we recommend three tiers. Read-only analysis runs unattended. Low-blast-radius writes, such as adding a comment to a catalog entry, need one human confirmation. Anything that changes a pipeline, a permission or a certified definition needs a named owner's approval and an audit entry.

What happens when one agent fails

Failure handling is where multi-agent designs usually get vague, so we define it per agent. If the Observability Agent times out, the Insights Agent still answers, but the response is flagged "freshness unverified" and cannot be used for a decision the user marked as high stakes. If lineage traversal fails, the answer drops its upstream claims and says so.

If the Catalog Agent fails, the risk is higher, because the Insights Agent would be guessing at definitions. In that case we block metric answers and allow only raw, clearly labelled table queries. A federated catalog also has a partial-failure mode: one source such as DataHub is down while Atlan responds. The Catalog Agent should return what it has with the missing source named, rather than a silent subset.

If the Insights Agent fails, the other two verdicts are still useful. The orchestrator can return the freshness and definition findings directly, which often answers the real question. "The orders table is eight hours late" is frequently the whole answer to "why did row counts drop."

Where this framework is limited

A fixed three-agent flow suits question answering and diagnosis. It is a poor fit for open-ended work such as redesigning a data model, where the roles are not this clean. It also depends on metadata quality: if your catalog has no owners and no definitions, the Catalog Agent returns thin context and the Insights Agent inherits the gap.

Observability coverage has its own limits. Vendor comparisons show how much coverage varies and is often priced by asset count; for example, Sifflet's comparison guide says its Entry tier supports up to 500 monitored assets. That is a vendor blog and the numbers are promotional, but it illustrates why the Observability Agent should be scoped to the assets the Catalog Agent has certified, not the whole warehouse.

If you want to inspect the metadata side yourself, the OpenMetadata repository is a good open-source reference for what a federated catalog source exposes.

Frequently Asked Questions

Why not let one agent do all three jobs? A single agent can, but it conflates detection with interpretation. When the same agent finds an anomaly and explains it, nothing independently checks the explanation. Separate agents give you separate verdicts you can audit, and each one can be swapped or upgraded without touching the others.

Do these agents need a shared memory? They need a shared context object per request, not a shared memory store. We pass asset IDs, verdicts and timestamps explicitly at each handoff. Long-term memory is a separate design question about what an agent should remember about your data.

Can I run this from Claude Code or Cursor? Yes. Each agent is an MCP server, so any MCP-compatible client can invoke it. The orchestration can live in the client session or in an Orchestration Agent, depending on how much you want to automate.

What should we build first? Start with the Catalog to Insights handoff, because wrong definitions cause the most damaging wrong answers. Add the Observability verdict second. Add approval-gated writes last, once the read-only flow has earned trust.