Don't ask the model to be brief. Cut it in code, and count what you cut.

2026-11-18来源 dev.toEnglish

DevReview, the AI code reviewer I'm building, never posts more than eight comments on a pull request [src:facts]. That limit isn't something the model is asked to respect. It's a constant in the application code, applied after the model has finished talking.

I've mentioned that design before, in one line: a prompt is a request, a slice() isn't. This post is the longer version of that line, because when I actually wrote down why, the reasons turned out to be more specific than "models don't always follow instructions." There is also one consequence I didn't expect: once the limit lives in code, the prompt has to say the opposite of what you'd normally write.

The obvious version, and why I didn't ship it

If you want an AI reviewer that doesn't bury a pull request in comments, the first thing you reach for is the prompt:

Only report the most important issues. Keep it to a handful of comments.

That works in the sense that the output usually gets shorter. For a demo, that's enough. I didn't keep it, and not because I measured it failing. I haven't measured how often a prompt-only limit gets ignored, and I don't want to invent a number for it. I dropped it because of what the limit is when it lives in the prompt, regardless of how well the model follows it.

There are four problems with it. Only the first one is about compliance.

1. A request has no failure mode you can see

When the limit is a sentence in the prompt, "the model followed it" and "the model mostly followed it" look the same until someone counts. If a review comes back with too many comments, nothing in the system notices. There's no exception, no log line, no failing check. You find out when a user complains that the tool is noisy, which is the exact complaint the limit was supposed to prevent.

It also drifts. A prompt that produces short output with one model, one set of sampling settings and one style of diff may not do the same after any of those change. You can re-test after every change, but what you're re-testing is a tendency, not a rule.

A limit in code has an obvious failure mode: the number of comments posted is greater than the constant. That can be asserted, and an assertion either passes or it doesn't.

2. The model chooses what to drop before anything is ranked

This is the one that actually decided it for me.

If the model is told to keep the list short, it does the trimming while it writes. Whatever it decided not to say never makes it into the output. So the selection, the most important decision in the whole review, happens inside the generation, in an order I don't control and can't inspect.

What I wanted instead was a fixed rule that I can read:

  1. Collect every finding.
  2. Sort by severity: BUG, then WARN, then NIT, then PRAISE [src:facts].
  3. Within the same severity, keep the order the model produced them in.
  4. Keep the top of the list up to the cap. Count the rest.

The rule isn't clever. Its value is that it's the same rule on every review, and when someone asks "why was this comment cut and that one kept?", the answer is a sentence, not a guess about what the model was weighing at the time.

A side effect is that the ranking is across the whole pull request, not per file. When you ask a model to be brief, you have no idea whether it's being brief across the PR or per file, or front-loading whatever it saw first. With a sort over the full list, a bug in the last file outranks a nit in the first one by construction.

3. You can't disclose what was never written

When DevReview cuts comments, it says so in the review: how many remarks were suppressed to keep the review focused, and (since a change I'll get to below) what severities they were. I call that the Noise Report.

That disclosure is only possible because the cut happens in code. The code has the full list and the shortened list, and the difference between their lengths is a number it can print.

If the model trims itself, there's no full list. The things it chose not to say don't exist anywhere. You can't count them, you can't show them, you can't tell the reader "there was more, and here's roughly what kind." A quiet review and a review that missed something look identical from the outside, and a prompt-side limit makes it impossible to tell them apart even in principle.

This is also why I don't describe DevReview as having nits "off by default." It doesn't. NITs are generated and kept like any other finding, and they're shown if they fit under the cap [src:facts]. They're just the first thing to go when the list is long, and when they go, they're counted.

4. The pull request is in the same context as your instruction

An AI reviewer reads text written by the PR author: the title, the description, the diff. All of that lands in the same context window as the instructions. If the limit is an instruction, then "keep it short" is one sentence competing with whatever else is in that window, including text written by someone you don't control.

I don't want to overstate this. The prompt tells the model to treat PR content as material to review, not as instructions, and that's the main defense. But for the specific question of "how many comments can end up on this PR," I'd rather the answer not depend on that defense holding. With the cap in code, the most any text in the PR can do to the count is fill it.

The part I didn't expect: the prompt has to say the opposite

Once the cap was in code, I had a prompt that still asked the model to be selective. That's two limits stacked on top of each other, and the combination is worse than either one alone.

Walk through it. The model trims itself to a short list, silently. The code then sees a list that's already under the cap, cuts nothing, and reports that nothing was suppressed. The review looks complete and disclosed, but the selection happened in the one place that can't be counted. The Noise Report says zero while the model has quietly dropped things. The code-side limit is still enforced, but the honesty it was supposed to buy is gone.

So the prompt now says the reverse. The relevant rule tells the model to report every genuine finding, and then:

DevReview enforces the comment budget in code — do not self-limit, but do not pad the list either. Only real findings.

Both halves matter. "Do not self-limit" is there so the code gets the full list and the count means something. "Do not pad" is there because the obvious way for "report everything" to go wrong is a model inventing filler to look thorough. The rest of the prompt pushes in the same direction: a list of common false positives to avoid, and an instruction to downgrade or skip anything it's not sure is a bug.

The split of responsibilities ends up clean:

  • The model decides what is a real finding and how severe it is.
  • The code decides how many get posted, in what order, and reports what it cut.

The model is good at the first job and has no reason to be trusted with the second.

What the code-side limit doesn't fix

Putting the limit in code moves the problem. It doesn't remove it. These are the costs and gaps I know about.

The ranking trusts the model's severity labels. The sort is deterministic, but the input to it isn't. If the model labels a real bug as a NIT, the code will faithfully rank it with the NITs and, on a long review, cut it. The code enforces the count, not the judgment. The best it can do is make the cut visible, so a reader who sees "suppressed: some NITs" at least knows there's something to expand.

The cap can cut a bug. If a pull request has more genuine bugs than the cap, some of them don't get posted. Sorting puts bugs first, so you see the worst of it, but "the cap never hides a real bug" is not something I can claim. I used to describe the suppressed comments as "lower-priority," and I removed that word for exactly this reason: when the list is all bugs, what gets cut is also a bug. The note now lists what severities were cut instead of characterizing them.

The order of filters decides what the count means. Counting suppressed comments is only honest if everything that removes a finding is counted, and if invalid findings don't compete for slots with valid ones. I got that wrong once: findings that pointed at files not in the PR were filtered after the cap, so they could take slots and then disappear, and in the worst case the review showed a green check. I wrote that up separately in When a Code Review Says "No Issues" but Found Bugs. The short version: the path check now runs before the cap, and discarded findings are counted too. A code-side limit doesn't make the pipeline honest by itself. It makes it possible to be honest, and then you have to check every filter.

You pay for findings you throw away. If the model reports everything and the code keeps a fraction, the tokens spent on the rest are spent for a number in the Noise Report. I think that's the right trade, since the count is part of what makes the review trustworthy, but it's a real cost and it grows with the size of the PR.

Why the cap is eight [src:facts] is a separate question. This post is about where the limit is enforced, not about its value. I don't have measured data on the value to share yet, so I'm not going to argue for it here.

Because it's code, it can be tested like code

The practical payoff of all this is that the limit stops being a behavior you evaluate and becomes a property you assert.

Testing a prompt-side limit means running the real model on a set of PRs and looking at the distribution of comment counts. That's useful, but it's an evaluation: it tells you how the model tended to behave on those inputs at that time.

Testing a code-side limit means faking the model's response and checking what the pipeline does with it. The model is out of the loop, so the result is the same every run. This is roughly the shape of the test (the helper name here is illustrative, not a file you'll find in the repo):

test("cap is enforced in code and overflow is counted", async () => { // Fake model output: more findings than the cap, severities mixed and // deliberately out of order, all pointing at files that exist in the PR. const findings = [ ...Array.from({ length: MAX_COMMENTS }, (_, i) => nit(`a.ts`, i + 1)), bug("b.ts", 10), warn("b.ts", 20), praise("a.ts", 30), ]; const result = await reviewWithFakeModel({ comments: findings }); // 1. Never more than the constant. expect(result.comments.length).toBeLessThanOrEqual(MAX_COMMENTS); // 2. Severity order wins over model order: the bug and the warning // the model listed last are posted first. expect(result.comments[0].severity).toBe("BUG"); expect(result.comments[1].severity).toBe("WARN"); // 3. Nothing disappears without being counted. expect(result.comments.length + result.suppressedCount).toBe(findings.length); }); 

Three assertions, and each maps to one of the reasons above:

  1. The count is a rule, not a tendency.
  2. Selection follows a fixed order that I wrote down, not one the model chose while generating.
  3. Everything that was cut is disclosed. If a future change adds a filter that drops findings without counting them, the third assertion is the one that should catch it.

The third one is the assertion I care about most, and it's the one a prompt-side limit can't have at all, because there's no "findings.length" to compare against when the model never wrote them.

Since the change I described in the post on the false green check, the review function has been tested end to end with faked responses from the model and from GitHub. The block above shows the idea, not a copy of that suite. The budget itself is also pinned by a plain unit test that feeds generated findings straight into the pure rank-and-slice function, so the cap, the order and the suppressed count don't depend on any model call at all. Before that, I relied on a demo pull request with an expected review written down before running it. One of its pass conditions was that the suppressed count equals the number of generated findings minus the cap. That's the same invariant as the third assertion, just checked by hand.

When a prompt-side limit is fine

None of this means "never put limits in prompts." I'd still use one when:

  • Nobody downstream relies on the limit. If longer output is just a bit annoying and nothing breaks, a prompt instruction is the cheapest thing that mostly works.
  • There's no list to cut. Free-form prose doesn't have a natural unit to slice. Truncating a paragraph in code is worse than asking for a shorter one.
  • You don't need to tell anyone what was left out. If there's no equivalent of the Noise Report, the strongest argument for doing it in code goes away.

For a code reviewer, none of those apply. The output is a list, the reader relies on it being short, and the reader also needs to know whether short means "nothing else" or "more, cut on purpose." That's what pushed the limit out of the prompt.

The rule I'd keep

If an LLM produces a list and something about that list has to be true every time (a maximum length, an order, a guarantee that nothing vanishes without a count), enforce it in code after generation and test it with a faked response. Use the prompt for the part only the model can do: deciding what goes on the list. Then make sure the prompt isn't quietly enforcing the same limit too, because a limit nobody can count is the one you were trying to get rid of.

I'm building DevReview at getdevreview.com. It's a GitHub App that ranks review comments by severity and caps them in code, and there's a free tier if you want to see what a capped review looks like on your own PRs. If you've solved the "the model trims itself before your code can count" problem differently, I'd like to hear how.

原文:https://www.a8q9.cn/en/article/don-t-ask-the-model-to-be-brief-cut-it-in-code-and-count-what-you-cut-b38d84.html

相关文章