Experiment Tracking by Hand vs With an AI Agent

Experiment tracking by hand means you record each run's config, code version, data version and metrics yourself, usually in a spreadsheet, a notebook or a naming convention. Experiment tracking with an AI agent means the agent captures those fields at run time and writes them to a tracker such as MLflow, so any two runs can be compared. Both approaches work. They differ in how often a field gets forgotten, and in who notices.

Key Takeaways

• •Manual experiment tracking fails at the handoffs: a config changed in a notebook but not logged, or a dataset version nobody recorded.
• •An agent's advantage is consistency. It logs the same fields on every run, whether or not the engineer remembers.
• •Our dw-experiment-tracking skill, run by the ML Agent, tracks runs with reproducible config, logged metrics and comparable lineage. We describe only that scope here.
• •Agents should not approve promotions, delete runs or change the metric that defines success. Those stay human decisions.
• •Evaluate an agent by asking it to reproduce a run from the record alone, then check whether the result matches.

Experiment tracking by hand vs with an AI agent: the job and who does it

The job is simple to state. When someone trains a model, the team needs to know afterwards what was run, with which code, on which data, with which parameters, and what it scored. Without that, a good result can't be repeated and a bad one can't be explained.

Today the person who trained the model does this job, which is usually a data scientist or ML engineer. In smaller teams it falls to whoever is closest to the notebook. The work is rarely anyone's main task, which is why it slips. Platform or data engineers get pulled in later, when a model in production behaves differently from the one that was evaluated, and someone has to work out why.

The manual workflow, step by step

We see the manual process vary a lot between teams, but it generally follows the same sequence. These are the steps, and where each one tends to fail.

• •Set parameters. Hyperparameters live in a notebook cell, a YAML file or command-line flags. Failure point: a value edited in place and never saved anywhere.
• •Pin the code version. The engineer notes a git commit, or doesn't. Failure point: uncommitted changes, so the commit hash doesn't describe what actually ran.
• •Pin the data version. A table snapshot, a file path or a query. Failure point: the path stays the same while the contents change underneath it.
• •Run training and record metrics. Numbers are copied from output into a sheet, a wiki page or a chat message. Failure point: transcription errors and missing metrics, especially for runs that were abandoned early.
• •Compare runs. Someone lines up rows by hand to decide which run was better. Failure point: runs that used different splits or metric definitions look comparable but aren't.
• •Hand off. A teammate or reviewer asks how a result was produced and the answer is reconstructed from memory.

None of these steps is hard. The problem is that each depends on a person remembering to do it, at the moment when they are most focused on the model rather than the record. Teams that adopt a tracker like MLflow Tracking remove the spreadsheet, but the engineer still has to call the logging code consistently and decide what to log.

What an agent does instead

The ML Agent is one of fifteen MCP-native agents in Data Workers. Its dw-experiment-tracking skill is described as: track ML experiment runs with reproducible config, logged metrics and comparable lineage. That is the scope we claim, and we keep this section to it.

• •Reproducible config: the run's parameters are captured as part of the record, not left in a notebook cell.
• •Logged metrics: results are written to the tracker for every run in the same shape.
• •Comparable lineage: each run carries enough context about its inputs that two runs can be compared on equal terms.

Because every Data Workers agent is an MCP server, an engineer can invoke this from Claude Code, Cursor or any MCP-compatible client, in the terminal they already use. A prompt such as "compare the last three runs of the churn model and tell us what differed" is the kind of question the record is meant to answer. The agent chains with our Quality and Observability agents, and we cover what that means for drift in a separate post. The MCP specification describes how clients and servers exchange these tool calls.

Manual vs agent: a side-by-side

We have deliberately left time-saved figures out of this table. We don't have a public benchmark for experiment tracking effort that we would stand behind, so we describe the effort qualitatively.

What still needs a human approval

Tracking is a record-keeping job, and a record is only useful if people trust it. We keep these decisions with humans, and the OSS edition of Data Workers is read-only by default, with writes and approval workflows in Pro.

• •Deciding what counts as success. The primary metric and its threshold are a team decision, not something an agent should pick.
• •Promoting a model. Moving a run to staging or production changes what customers experience.
• •Deleting or overwriting runs. The history is the point; removal needs a person and an audit trail.
• •Changing the evaluation set. A new split makes old and new runs incomparable, so someone has to own that call.

How to evaluate an agent for this job

We suggest testing an agent the way you would test a new colleague. Don't ask whether it sounds right. Ask whether the record it produces survives scrutiny.

• •Reproduce a run. Pick one logged run, give someone only the record, and ask them to rerun it. Compare the metrics. Small floating-point differences are expected; large ones mean something wasn't captured.
• •Break a run on purpose. Change a parameter in a notebook without saving it and see whether the record reflects what actually ran.
• •Check the gaps. Abandon a run halfway and see whether the agent logs it or silently drops it.
• •Read the access model. Confirm which operations are read-only and which write, and where approvals sit.
• •Look at where the data goes. Confirm the record stays inside your infrastructure. Data Workers runs in the customer's environment.

For a wider view of how MLflow-style tracking fits into the rest of an ML platform, the MLflow documentation is the authoritative reference, and our agents work with that kind of tracker rather than replacing it.

Where this fits in a data platform

Experiment tracking rarely fails in isolation. A model's score drops because a feature table changed, and the feature table changed because an upstream schema moved. Our Schema Agent maps that blast radius, and our Catalog Agent) keeps lineage current, so a run's inputs can be traced back to a table someone owns. We covered how Data Workers keeps the catalog current in a separate post. That cross-agent context is the reason we think about tracking as part of the platform, not a standalone notebook habit.

Frequently Asked Questions

Is experiment tracking with an AI agent more accurate than doing it by hand? It is more consistent, not smarter. The agent logs the same fields on every run, so gaps from forgetfulness shrink. The quality of the record still depends on what you ask it to capture.

Do we need to replace our existing tracker? No. The point of the ML Agent is to work with the tooling you already run, such as MLflow or Weights & Biases, so the record lands where your team already looks.

What can the dw-experiment-tracking skill do? Its description is to track ML experiment runs with reproducible config, logged metrics and comparable lineage. We don't claim more than that for this skill.

Can an agent decide which model to promote? We recommend against it. Promotion and the definition of success should stay with people, with an approval step and an audit trail.

How do we know the agent's record is trustworthy? Test it by reproducing a run from the record alone. If the metrics match, the record captured what mattered. If they don't, you have found a gap before it mattered in production.