I Saved Every AI Code Review Message for a Sprint. Here's What Stood Out.
I Saved Every AI Code Review Message for a Sprint. Here's What Stood Out.
For one sprint I archived every message from our AI code reviews. Full transcripts, not summaries. One agent writes the change, a second reviews it, and they go round after round until the review passes or a human steps in. Then I read all of it. Going through those logs taught me more than skimming merged PRs ever did.
Rejections carried most of the value
Pass reviews are short and boring. The reviewer runs its checks, finds nothing, signs off. All the interesting reading lived in the rejection rounds. A rejection forces the author agent to restate what the change actually does, and halfway through restating it, the author sometimes catches a second problem on its own. In our logs, several bugs got fixed while answering a rejection, before the reviewer even looked again.
One catch I kept rereading
The find of the sprint was a download function with no failure handling. Server returns an error page, the code saves that HTML as the downloaded file, and nobody notices until a user opens it. The reviewer agent flagged it in round one. Reading the transcript back, the rejection message does not just name the bug. It points at the exact spot where the response body gets written unconditionally.
Review comments that stop evaporating
Our review conversations live in loops, the workspace unit that carries a task from kickoff to acceptance. Every round of comments stays attached to the loop, so when a change lands three weeks later I can open it and read why an early decision was made. Before this, review feedback lived in chat and evaporated. Searching chat for old feedback is archaeology with unreliable results.
Feedback that rolls forward
The part that surprised me most sits at the end of the process. Rejections and the final acceptance feed preference cards. Each card is one verifiable rule with the evidence behind it, where it applies, and a confidence level. The download bug became a card about handling non-200 responses in any code path that writes files. The next task of that shape starts with that card already in context, so the lesson shows up before I have to repeat it.
Two or three rounds, then a human
Most PRs settled around two or three rounds. A single round usually meant the change was trivial. Past four rounds, something was off in the task itself, a vague spec or a wrong assumption baked in from the start. Those went to a person. Looking back, round count tells you about spec quality as much as code quality. Those numbers are our experience over one sprint, not a benchmark.
If you only read merged PRs, you see the final state, which is the least informative part of the process. The messages before it, especially the rejected ones, show what your review setup actually catches versus what it waves through.
All of this runs on Octo, the open-source workspace we use for human-agent collaboration. The review setup is its Critic mode, where the author agent and the reviewer agent never see each other, and work moves through do, review, reject or pass. Nothing else crosses between them. The loops and preference cards are what make the transcripts worth keeping.