Streaming Lineage: By Hand vs With an AI Agent

Tracing streaming lineage by hand means reconstructing the path from producers through Kafka topics and stream processors to consumers and sinks, usually from several systems that each hold one piece of the answer. An agent can do the same trace by querying those systems through MCP tools and returning one assembled graph. Confluent documents the same goal in Stream Lineage, and OpenLineage defines an open event format for it. This post compares the manual and agent-driven trace, and what you should still approve yourself.

Key Takeaways

• •Streaming lineage is hard to do by hand because the facts are spread across broker metadata, a schema registry, processor configs and consumer groups, and no single system holds the whole path.
• •The manual trace is a read-only investigation, which makes it a good candidate for an agent; the risky steps are the ones that change something, not the ones that look.
• •Kafka's default log retention is 168 hours, so a hand trace based on what is currently in the topic only sees recent traffic, not the full history of who used it.
• •An agent should return the graph with its evidence and mark the hops it could not confirm, so a reviewer can check each edge.
• •Evaluate an agent for this job on edge coverage, evidence per edge, handling of unknown hops, and whether writes are gated behind approval.

The job and who does it today

The job is to answer a question like "what breaks if I change this topic's schema?" or "where does this PII field end up?" For streaming systems, the answer is a graph: producers write to topics, processors read and write topics, and consumers and sinks take the data out to warehouses, search indexes and services. The graph has to be current, because topologies change whenever a team ships a new consumer.

Today the person doing this is usually a data or platform engineer who owns the Kafka cluster, or an analytics engineer who got a ticket about a downstream dashboard. They are rarely the person who built every producer or consumer, so the trace is partly archaeology. The same trace is needed for incident response, schema-change reviews, access audits and decommissioning old topics.

Batch lineage tooling often starts from SQL and orchestration metadata. Streaming adds a layer those tools do not read natively: the broker, the registry and the processor runtime. That is why teams often end up with a warehouse lineage graph that stops at the ingestion boundary, and a separate, undocumented picture of what happens before it.

The manual workflow, step by step

A typical hand trace has five steps. First, list the topic's producers, usually from client IDs, ACLs, service accounts or whatever the application repos configure. Second, read the schema from the registry and note the compatibility mode and the subject history. Third, find the stream processors, such as Flink jobs, Kafka Streams apps or ksqlDB queries, that read the topic and write others. Fourth, list consumer groups on the topic and map each group back to an owner. Fifth, follow the connectors and jobs that write to sinks.

Each step uses a different tool and often a different person. Broker metadata comes from the Kafka CLI or admin API. Schemas come from the Schema Registry. Processor topology lives in job configs and repositories. Sink mappings live in Kafka Connect configs or in application code. The engineer copies results into a document or diagram and repeats the loop for every topic the processors write to.

Retention makes this harder. Kafka's broker configuration docs list a default log.retention.hours of 168, which is seven days. A consumer group that reads a topic once a month, or a batch job that replays on demand, may leave no trace in the current data. The trace then depends on group metadata, access logs and the memory of whoever owns the job.

Where the manual trace breaks

The first failure is staleness. A diagram drawn during last quarter's migration does not show the consumer a team added last week. The second is silent gaps: a service that reads through a custom client with no registered group name looks like nothing in broker metadata. The third is fan-out. One topic feeding a processor that writes five topics means five more traces, and the effort grows with the graph rather than with the question.

A fourth problem is that nothing proves the trace is complete. The engineer stops when the story seems coherent, not when every hop is confirmed. Emitting lineage events from the runtime is the more durable fix, and the OpenLineage Flink integration is one example of that approach. But instrumentation only covers the jobs that are instrumented, so teams still need a way to find and report the rest.

Finally, the handoffs cost time. The platform team knows the brokers, the application team knows the consumers, and the governance team knows which fields are sensitive. A single question can bounce between three people before anyone has the whole picture. That is the part of the work an agent is most suited to take over.

What an agent does instead

The Data Workers skill dw-streaming-lineage, part of the Catalog Agent, is described as tracing lineage across the streaming topology: producers, topics, processors, consumers and downstream sinks. In practice that means the agent runs the same five lookups an engineer would, through tools exposed over the Model Context Protocol, and assembles the results into one graph instead of leaving them in five terminals.

The useful property is not speed on a single topic. It is that the agent repeats the same procedure for every hop, records where each edge came from, and can say which hops it could not confirm. A reviewer then reads a graph with evidence attached, rather than a diagram with no provenance. We would treat any unconfirmed hop as an open question for its owner, not as an absence of a dependency.

Because the Catalog Agent) federates across catalogs, the streaming graph can be joined to the warehouse side, so a question about a topic can continue into the tables it feeds. We are describing the skill as documented, not claiming it finds every consumer in every environment. Coverage depends on which systems are connected and what metadata they expose.

Manual vs agent, side by side

What still needs a human approval

Reading is the safe part. Per our product description, the open-source edition is read-only by default, and writes, approval workflows and audit trails are part of the Pro tier; see dataworkers.io for the current scope. For lineage, this means the agent can trace and report without touching your cluster, and any step that would change a topic, an ACL or a connector should wait for a person.

Several decisions stay with people regardless of tooling. Deciding that an unowned consumer group can be deleted is a business call. So is accepting a breaking schema change after seeing who reads the topic, or declaring a sink out of scope for a PII review. The agent supplies the evidence for these decisions, and the owner of each system makes them.

We recommend a simple rule: the agent may propose, the owner approves, and the approval is recorded. If a vendor cannot show you that separation, the agent is not ready to run near a production cluster, whatever it can do in a demo.

How to evaluate an agent for streaming lineage

Start with a topic whose lineage you already know well, and a second one you do not. Ask the agent to trace both. On the known topic, count how many true edges it finds and whether it invents any. On the unknown topic, check a sample of edges against the source systems. The point is to measure precision and coverage on your topology, not to trust a demo dataset.

• •Edge coverage: does it find every producer, processor, consumer and sink you can independently confirm?
• •Evidence per edge: can you see which system and which record each hop came from?
• •Unknown handling: does it list hops it could not confirm, or fill the gap with a plausible guess?
• •Freshness: does a new consumer show up on the next run without manual updates?
• •Permissions: are credentials read-only, and are write actions gated behind approval?
• •Portability: does it work from the MCP client your team already uses, and can its output be exported in an open format such as OpenLineage?

Finally, ask what happens when a source is down or returns partial data. A good answer is a partial graph with the missing sources named. A poor answer is a clean graph that quietly leaves them out.

Frequently Asked Questions

Is streaming lineage the same as batch lineage?

No. Batch lineage is typically derived from SQL, orchestration and warehouse metadata, while streaming lineage also has to cover brokers, schema registries, processors and consumer groups. The two should be joined at the sink so a question can follow the data from producer to dashboard.

Does the agent change anything in my Kafka cluster, and is it safe to run in production?

Tracing is a read operation. Per our product description, the open-source edition is read-only by default, so it can inspect metadata without altering topics or ACLs. Give it read-only credentials regardless, and require approval for any change a future workflow proposes.

Can an agent find consumers that never registered a group?

Only if some connected source records them, such as access logs or network metadata. If no source does, the agent cannot know about them, and a good one will tell you which parts of the topology it could not see.

Do I still need OpenLineage if I use an agent?

They solve different parts of the problem. Runtime lineage events record what jobs actually did, while an agent assembles and queries lineage across sources, including ones that are not instrumented. Teams can use both and compare what each finds.