Would Your AI Reviewer Catch a Bug You Planted?

An AI reviewer approves a change. I want to know what would make that same review process reject it—and whether the rejection would actually stop the merge.

In our discussion about independent AI review, Glen Allen suggested seeding defects to understand which kinds of problems different reviewers detect.

That suggests a useful next step: give a review workflow a small, controlled opportunity to be wrong.

This is a proposed test design. I have not run this comparison yet.

Start with a known-good candidate

Consider a fictional task: update a function under src/**, preserve its documented support for empty input, and review the exact candidate before acceptance.

The test owner first establishes a baseline that satisfies those requirements. Record its commit or content digest, the task specification, and the evidence available to each reviewer.

Then create isolated variants, changing one condition at a time:

The test owner records expected outcomes before the run. Reviewers receive the requirements and their normal evidence access, with the injected locations and case labels withheld.

The control matters: it exposes a workflow that rejects every candidate. Passing this exercise establishes only the specified checks; release authorization remains a separate decision.

Record the detector and the consequence

For each run, I would record:

• The exact candidate and the defect introduced.
• Which component detected it: an AI reviewer, a test, a scope check, or an artifact-binding check.
• The finding and the evidence supporting it.
• The acceptance decision and which subsequent action was blocked.
• Anything the observer could not establish.

This makes the result interpretable. A scope guard catching an unauthorized path is evidence about that guard. A reviewer reproducing the empty-input failure is evidence about that review. An infrastructure timeout leaves detection unestablished.

Detection and enforcement also need separate observations. A precise warning is useful, but the exercise should show whether the workflow actually withheld acceptance because of that warning.

Keep the experiment separate from the candidate

In the earlier discussion about interrupted agents, Jackson Ly described a reviewer leaving an intentional test mutation in the working tree after reaching a usage limit. That is his reported experience; I have not independently reproduced it.

I would use disposable copies, keep the baseline unchanged, and verify its identity after the experiment. Any attempted merge or publication would target a test destination.

Each result supports a claim about that particular case and setup. Broader conclusions about independence or reliability would require additional cases and evidence about shared failure sources.

One case to bring to the discussion

What is one deliberately broken change your review process must catch, and what evidence would show it was caught?

A short example with the expected detector and stopping point would help make the next comparison concrete.

Disclosure: I maintain Agent Role Contracts (ARC) at Netsujo. ARC checks the consistency of supplied role, scope, review, and handoff declarations. Actual permissions, reviewer behavior, and acceptance enforcement require controls in the surrounding system. This proposed experiment concerns those controls.