Count outcomes, not agents

When organisations start working with AI agents, progress is often reported as a count of agents built. The count is easy to measure, but it says little about whether the work has improved.

Measure the work, not the agents. Before any agent goes live, record three or four measures for the process it will touch. Good candidates are:

• cycle time;
• rework or error rate;
• cost per case;
• customer satisfaction.

These measures become the baseline.

Pilot one step. Choose a single step in the process. Let an agent draft or prepare the work, and have people review every case. This keeps quality visible from the first day.

Compare like with like. After a set period, take the same measures again. If the step is faster, more accurate or cheaper, you have evidence. If it isn't, you have learned something useful at low cost.

Count the full running cost. The business case should include:

• model usage;
• the time people spend reviewing;
• monitoring;
• the effort of changing the process.

A pilot that looks inexpensive can be costly to run at scale, and the reverse can also be true.

Fund in stages. Release the next budget when the last stage has shown its value. This keeps decisions grounded, and it gives finance, operations and technology teams a shared view of progress.

A short checklist:

• Do you have a baseline for the process before the agent arrives?
• Is every pilot reviewed by people until the evidence is in?
• Does your business case include the cost of review and monitoring?

From the 2SD Technologies team (2sdtechnologies.com). Part 4 of "Don't just build agents".