How to Measure Toil in a Data Team Before Automating It

To measure toil in a data team before automating it, start from the definition in the Google SRE book chapter on eliminating toil: work that is manual, repetitive, automatable, tactical, without enduring value, and that grows linearly with the size of the service. Then log two weeks of real interrupt-driven work, tag each item against those criteria, and rank by hours per week. Automate only what scores high on all of them.

Key Takeaways

• •Toil is a specific category of work, not a synonym for "annoying." The SRE book defines it by six properties, and a task needs most of them to qualify.
• •Measure before you automate: a two-week time log of interrupts beats a team survey, because people remember the painful tasks, not the frequent ones.
• •Rank candidates by hours per week multiplied by how automatable the task is, not by how much engineers dislike it.
• •Some data work looks like toil but is judgment work (metric definitions, data modeling, access exceptions). Automating it badly creates incidents.
• •The SRE book sets an aim of keeping toil under 50% of an engineer's time. We treat that as a useful ceiling for data teams too, and we say plainly that it is borrowed from SRE, not a data-specific benchmark.

How to Measure Toil in a Data Team Before Automating It: The Short Method

Most data teams jump straight to tooling. Someone is tired of rerunning a failed DAG, so they buy an observability product or write a retry wrapper. Sometimes that is right. Often the thing eating the team's week is something nobody named, such as answering "why did this number change?" in Slack six times a day. You cannot find that without measuring.

The method below takes about two weeks of light logging and one hour of analysis. It needs no special tooling. A shared spreadsheet or a Jira label is enough. The five steps follow.

Step 1: Define toil for your team using the SRE criteria

Agree on a definition before you log anything, or every annoying task will end up on the list. The SRE book lists the properties of toil: manual, repetitive, automatable, tactical (interrupt-driven rather than strategy-driven), no enduring value, and O(n) with service growth. For a data team, translate each one:

• •Manual: an engineer runs a query, a script, or a click-path by hand each time.
• •Repetitive: it has happened at least three times in the logging window with the same shape.
• •Automatable: a machine could do it given the right context and permissions.
• •Tactical: it arrived as an interrupt (an alert, a Slack question, a ticket), not as planned roadmap work.
• •No enduring value: after finishing it, the system is no better than before.
• •Scales with growth: more tables, more pipelines, or more stakeholders means proportionally more of it.

Write these on the top of the log sheet. If a task fails the "repetitive" or "automatable" test, it is not toil, however much it hurts.

Step 2: Log every interrupt for two weeks

Ask each engineer to add a row whenever they stop planned work to handle something. Keep the schema small so people actually fill it in: date, short description, who asked or what triggered it, minutes spent, and the system touched. Two weeks covers at least one full pipeline cycle and one month-end or reporting spike if you time it right.

Do not rely on a survey instead. People overweight rare, dramatic incidents and underweight the ten-minute tasks that happen forty times. Your ticketing system and on-call history are useful cross-checks, but they undercount Slack and hallway requests, which is where much data toil lives.

Typical categories we see teams end up with, based on our own conversations and not a survey we can cite:

• •Failed or late pipeline runs: triage, rerun, root-cause lookup.
• •Schema drift: a source column changed and downstream models broke.
• •"Why did this number change?" questions from analysts and finance.
• •Access requests and PII audits: who can see what.
• •Documentation and catalog upkeep: descriptions, owners, tags.
• •Warehouse cost questions and cleanup of unused tables.

Step 3: Tag each item against the toil criteria

At the end of the window, add a column for each criterion and mark yes, partly, or no. This takes about an hour for a team of five. The goal is not precision. It is to separate real toil from judgment work and from one-off projects that happened to land as interrupts.

A quick rule of thumb we use: if you cannot write the task as a runbook with fewer than ten steps that never ask the engineer for an opinion, it is probably not ready to automate. It may still be worth assisting with tooling, but that is a different decision.

Step 4: Score and rank the candidates

Group log rows by task type and total the minutes per week. Then multiply by an automatability score you assign: 1 for needs human judgment, 2 for a machine could draft and a human approves, 3 for a machine could do it end to end. The table shows an illustrative example with made-up numbers to show the mechanics. Replace them with your own.

The score is crude on purpose. Its job is to stop the loudest complaint from setting the roadmap. In the example, defining a new metric takes real time but scores low because it needs judgment and is not repetitive. That is design work, and you should protect time for it, not automate it.

Step 5: Decide what to automate, assist, or leave alone

Split the ranked list into three buckets. Automate means a deterministic script, a dbt test, or a scheduler policy can handle it with no human in the loop. Assist means an agent or tool does the investigation and proposes the fix, and a human approves. Leave alone means the task needs judgment, or happens too rarely to repay the build cost.

The "assist" bucket is where AI coding agents change the math. Work such as root-cause tracing across lineage, logs, and query history used to be too variable to script. An agent connected over the Model Context Protocol can read the same systems an engineer would and return a diagnosis. Our Incidents Agent does this for failed pipelines, and our Schema Agent maps the downstream blast radius of a column change. Both run inside Claude Code, Cursor, or any MCP-compatible client, so the engineer asks the question where they already work. The OSS edition is read-only by default, which suits the assist bucket: it investigates and proposes, and a human decides.

We are the vendor here, so apply the same scoring to us. If your top toil item is access-request approvals, an agent that only reads and proposes will not remove the approval step, and it should not. If your top item is something rare and high-stakes, a runbook may be cheaper than any tool.

What Measuring Toil Does Not Tell You

Hours spent is a proxy. A task that takes ten minutes but blocks the finance close has more business cost than its minutes suggest. Add a severity column if your team has tasks like that. Likewise, toil that lands on one person burns that person out faster than the same hours spread across five. Look at the distribution, not only the total.

Also be careful about measuring the same team twice. Once you announce an audit, behavior shifts: people stop taking small requests, or log more generously. Run it again after three months of automation work using the same method, and compare like with like. That second measurement is what tells you whether the automation paid off, which is the part most teams skip.

Where to Go Next

If the audit points at schema drift or broken pipelines, we covered the mechanics of how an agent traces root cause in our posts on autonomous data operations and the context layer. If governance requests top your list, our Governance Agent page explains how PII detection and audit trails are generated. Once you have a ranked list, pick one item from the "assist" bucket, run it for a month, and re-measure before adding another.

Frequently Asked Questions

What counts as toil in a data team? Toil is manual, repetitive, automatable, interrupt-driven work that leaves no lasting improvement and grows as the platform grows. Rerunning failed jobs, hand-checking schema changes, and answering the same lineage question repeatedly all qualify. Designing a data model or deciding a metric definition does not.

How long should a toil audit run? Two weeks is the minimum we recommend, because it captures at least one full pipeline cycle. Four weeks is better if your team has monthly reporting peaks. Shorter windows overweight whatever incident happened that week.

What percentage of time should a data team spend on toil? There is no data-specific benchmark we can cite. The SRE book states an aim of keeping toil below 50% of an engineer's time, and many teams use that as a starting ceiling. Treat it as a reference point from a different discipline, and set your own target after your first measurement.

Should we automate everything the audit flags? No. Automate deterministic tasks, assist with tasks that need investigation, and leave judgment work alone. Automating a task that needs human context, such as granting access exceptions, tends to create incidents instead of removing them.

Do we need an AI agent to reduce toil? Not necessarily. Many high-scoring items are solved by a dbt test, a scheduler retry policy, or a runbook. Agents are most useful for variable investigation work, like tracing why row counts dropped overnight, where the steps differ each time but the sources of evidence are the same.

How do we know the automation worked? Repeat the same two-week log after three months and compare hours per task type. If the hours did not fall, either the automation missed the real cause or the work moved to a different task type. Both are useful findings.