Kafka Stream Config by Hand vs With an AI Agent

An AI agent can take over the repetitive parts of Kafka stream configuration (choosing partitions, retention, replication and per-topic overrides) but it should not apply them to production without a human approval. The Apache Kafka documentation defines the settings that matter, and the manual workflow is mostly about applying those settings consistently across many topics. That consistency is the part an agent handles well. The decision to apply a change is the part that stays with an engineer.

Key Takeaways

• •Manual Kafka config is slow mainly because of review and drift, not because any single command is hard.
• •An agent is useful for proposing safe defaults, explaining tuning trade-offs, and diffing a proposed config against what is live.
• •Partition count can be increased but never decreased, so partition changes always need a human approval.
• •Durability comes from combining replication factor, min.insync.replicas and producer acks, not from any one setting.
• •When evaluating an agent, test whether it shows a diff before it writes anything, and whether it works through an approval step.

Kafka stream config by hand vs with an AI agent: the job and who does it

In most data teams, one or two platform or data engineers own Kafka topic configuration. A product team asks for a new topic, and a platform engineer decides partitions, replication factor, retention, cleanup policy and compression. Later someone else is paged when disk fills up or a consumer falls behind. The job looks small per topic, but it repeats for every new stream and rarely gets revisited.

This post is for the person deciding whether an agent should do part of that work. We make the Streaming Agent, so we have a stake, and we say where the manual path still wins.

The manual workflow, step by step

The typical hand-run process has six steps, and each one is a handoff.

• •Gather requirements: expected throughput, key cardinality, how long consumers need data, whether the topic is a changelog or an event stream. This usually happens in a ticket or chat thread.
• •Pick partition count: this sets the ceiling on consumer parallelism. Kafka allows increasing partitions later, but not decreasing them, and increasing them changes which partition a given key maps to.
• •Pick replication and durability: replication factor, min.insync.replicas, and the producer's acks setting have to agree. A replication factor of 3 with min.insync.replicas of 2 and acks=all is the commonly documented durable combination.
• •Pick retention and cleanup: retention.ms or retention.bytes for delete policy, or cleanup.policy=compact for keyed state. The broker default retention is seven days, which is often wrong for both audit topics and high-volume firehose topics.
• •Create the topic with kafka-topics.sh or infrastructure code, then apply overrides with kafka-configs.sh.
• •Review and monitor: someone checks the change, and later someone notices if disk, lag or under-replicated partitions drift.

Where the manual path breaks

The commands are not the hard part. The failures we see described most often are these:

• •Drift: topics created through a console, a script and Terraform end up with different overrides, and nobody can say which is canonical.
• •Silent mismatch: min.insync.replicas equal to the replication factor means one broker restart blocks writes, while a value of 1 with acks=all gives weaker durability than people assume.
• •Retention guesses: retention is set once from a rough estimate and never compared to actual consumer lag or disk growth.
• •Irreversible partition choices: because partitions cannot be reduced, an over-provisioned topic stays over-provisioned.
• •Context loss: the reason a topic has a 30-day retention lives in a closed ticket, so the next engineer is afraid to change it.

None of these are Kafka bugs. They are process problems at scale, which is where automation helps.

What an agent does instead

We scope the agent to what its skill, dw-kafka-stream-config, actually describes: configuring Kafka streams, topics, retention and replication with safe defaults and tuning guidance. In practice that means an engineer asks in Claude Code or Cursor, and the agent responds through MCP. The Model Context Protocol is the open standard that lets those clients call tools such as this one from the terminal.

• •It proposes a topic config from stated requirements, starting from safe defaults rather than broker defaults.
• •It explains the trade-off behind each setting, so the reviewer sees why a value was chosen.
• •It compares a proposed config against an existing topic and shows the difference before any change.
• •It keeps the rationale next to the config, which addresses the context-loss problem.

The OSS edition (dw-claw, Apache 2.0) is read-only by default, so it can inspect and recommend but not change a cluster. Writes, approval workflows and audit trails are part of Pro. We say this plainly because many teams should start in the read-only mode and judge the recommendations before enabling writes.

Manual vs agent: side-by-side comparison

The manual path is better when you already have well-tested Terraform modules and a small topic count. A wrapper around kafka-topics.sh is cheap, and an agent adds little there. The agent earns its place when topic count is high, ownership is spread across teams, or reviewers lack Kafka depth.

What still needs a human approval

We recommend the following changes always pass through a person, whether an agent proposed them or not:

• •Any partition increase, since it changes key-to-partition mapping for keyed topics.
• •Lowering retention, since it can delete data consumers still need.
• •Changing cleanup.policy from delete to compact or the reverse.
• •Changing replication factor on a live topic, which triggers partition reassignment and extra broker load.
• •Any write to a production cluster.

Pro enforces this through approval workflows and tamper-evident audit trails, so the approver and the change are both recorded. Schema changes on the same stream are a separate concern; our Schema Agent covers blast radius there, and we covered how agents coordinate across tools in other posts on this site.

How to evaluate an agent for this job

Run these checks on any agent, ours included, before trusting it with a cluster:

• •Does it show a diff before changing anything, and can you run it with writes disabled?
• •Does it explain why a value was chosen, or only output a number?
• •Does it flag inconsistent durability settings, such as min.insync.replicas equal to the replication factor?
• •Does it refuse or escalate irreversible changes like partition reductions, which Kafka does not support at all?
• •Can you audit who approved a change and when?
• •Does it work inside the client your engineers already use, such as Claude Code or Cursor, rather than in a separate dashboard?

A useful pilot is to point the agent at ten existing topics in read-only mode and compare its recommendations to what your best Kafka engineer would choose. If the two disagree, read the reasoning; sometimes the agent is wrong and sometimes the topic really has drifted. We treat that disagreement as the useful output.

Frequently Asked Questions

Is it safe to let an AI agent change Kafka topic configs? It is safe only with guardrails. We recommend read-only mode first, a visible diff before any write, and human approval for partition, retention and replication changes.

Can you reduce the number of partitions on a Kafka topic? No. Kafka only supports increasing partitions, as described in the Kafka documentation. Reducing means creating a new topic and migrating consumers, which is why partition choices deserve review.

What replication settings make a Kafka topic durable? The commonly documented combination is a replication factor of 3, min.insync.replicas of 2, and producers using acks=all. All three must agree, because any one alone does not give the guarantee.

Does the agent replace Terraform or other infrastructure code for Kafka? No. In our view the agent proposes and explains configuration, and the approved result should still land in whatever source of truth your team uses.

Which clients can run the Streaming Agent? Every Data Workers agent is an MCP server, so it works in Claude Code, Cursor, Windsurf, VS Code, GitHub Copilot, or any MCP-compatible client.