Track the Edit Rate: How We Decide How Much to Trust Each AI Agent Task
The question I get most about running AI agents in a small company is some version of "can you trust it?" I think that's the wrong question. You don't trust an agent. You trust an agent on a specific task, for now, based on what the review data says.
I'm Justin Pennington, founder of Infraxio, based in Ponte Vedra in St. Johns County, just south of Jacksonville, Florida. We're a seven-person company, down from about thirty before AI, and agents do real work inside our products every day. This is a builder-oriented rewrite of an essay on my site, AI Agents Do Not Manage Themselves. More about me and what we're building: infraxio.com/justin-pennington.
Here's the short version: the agent isn't the system. The agent plus the review loop is the system. If you skip the loop, you end up with one more tool running in the background producing output nobody trusts.
Put agent output where human output already goes
The first design choice that mattered for us: agent-completed work lands in the same review column as human-completed work. In IFX Hub, when an agent drafts a client summary, assembles data for an invoice, or takes a first pass at a bid section in IFX Bid, it shows up on the board right next to everything else.
That sounds minor. It isn't. If reviewing agent output means opening a separate dashboard, reviews quietly stop happening. If it's already in the queue the team checks every morning, it gets read.
It also gives you an audit trail for free. Every agent task carries the same record a person's task does:
- who assigned it and when
- what the agent produced
- who reviewed it
- what the reviewer changed
That last field is the one worth building everything around.
Trust is per task, not per agent
Early on I treated trust as a yes-or-no thing about the agent. In practice it splits by task type.
The same agent, on the same model, can be very reliable at turning structured data into an internal status update and much less reliable at writing the narrative section of a bid response. The first is close to deterministic: inputs map cleanly to outputs. The second takes judgment about emphasis and tone.
So instead of one trust setting, we keep a review level per task type:
| Review level | What the reviewer does | Typical fit |
|---|---|---|
| Light | Quick scan that the output matches the expected pattern | Structured-data summaries, internal updates |
| Heavy | Reads closely and rewrites parts | Narrative or client-facing writing |
| None (human only) | The agent doesn't get the task | Anything where one bad output costs more than the time saved |
Tasks move between levels. One that starts on heavy review can drop to light after a run of clean output. One on light review goes back to heavy after a miss. There's no final state. It's ongoing calibration.
The numbers that tell you whether it's working
Because every review records what changed, you can ask operations questions instead of guessing:
- How often does agent output ship as-is versus get edited? That's the edit rate, per task type.
- Which task types have the highest edit rate? Those are your candidates for heavier review, a better input, or handing the task back to a person.
- Where is review taking longer than just doing the work? That's a net loss, however impressive the demo looked.
The math is plain. An agent that saves four hours a week on drafting and costs two hours of review is a win. One that saves an hour and costs three hours of correction is a loss, and it needs to improve or lose the task.
At thirty people, a bad process could hide because someone quietly absorbed the extra work. At seven, if review overhead isn't paying for itself, someone notices by Wednesday. That visibility is an advantage if you actually look at the data.
The failures that matter look plausible
The obviously wrong outputs aren't the problem; anyone catches those. The dangerous ones look right:
- a project summary that's accurate except for one milestone the agent never saw, because the update came through a channel it doesn't watch
- an invoice draft with the right line items and the wrong rate, because the price changed in a conversation last week and never made it into the system
This is why it's tempting, and wrong, to stop reviewing a task after twenty clean runs. The twenty-first is where a scope shift or an off-system pricing conversation slips through.
These are the same mistakes a new hire would make. The difference is in the fix. A person learns from the correction. An agent makes the same class of error next time unless you change its input, its prompt, or its scope. With a person you coach. With an agent you adjust the system.
Fix the data before you blame the model
An agent working from dirty data doesn't stop and say "this looks off." It processes what it sees with the same confidence either way.
That's why consolidation matters more once agents are doing work. When a deal, its delivery tasks, and its billing live in one place, the agent sees the whole picture. When they live in three systems, either the agent sees one slice or you build integration logic to stitch the context together, and that integration layer becomes one more place for data to go stale.
If you're adding agents, plan for this
- Route agent output into the review queue your team already uses.
- Record what the reviewer changed, not just whether they approved.
- Set a review level per task type and expect it to move.
- Watch edit rate and review time, and take tasks away from agents that don't earn them.
- Clean up the system of record before you scale the agents reading from it.
Agents don't manage themselves. Neither did the people they replaced. The management just looks different now.
I'm curious how other teams handle this. Do you track something like an edit rate on agent output, or is review still mostly gut feel?