My agent's verifier rejected 0 of 11 answers, and the score still dropped

I added a verify-before-finish gate to a ReAct agent: before the loop accepts a final answer, a second model call checks whether the trajectory supports it. On 15 two-hop questions with Gemini 3.1 Flash-Lite, plain ReAct solved 10 and ReAct with the gate solved 9. In my results file I explained the drop as the price of verification: the extra check eats into a fixed iteration budget. (That paragraph is corrected now.)
That explanation was wrong. The verifier ran 11 times and accepted all 11 answers. It never rejected anything, so it never cost the loop a step. The gap came from somewhere else, and so did most of the failures.
This is from agentic-patterns, where I implement agent patterns in plain Python and measure them against live models.
The setup
The benchmark uses an invented knowledge base of 20 facts about rivers, mountains and countries that don't exist, so the model can't answer from memory. Every question takes two hops. "What is the capital of the country the Kelvorin River flows through?" needs the river's country first, then that country's capital. The agent writes Thought/Action text, has search and lookup tools, and gets 6 steps.
The verify variant runs the same loop with one sentence appended to the system prompt:
system_prompt = ( f"{FEW_SHOT_SYSTEM_PROMPT}\n\nNote: a verifier checks every Finish against the trajectory " "before accepting it, so only finish once your observations actually support the answer." ) The docstring gives a second reason for that sentence: it keeps the two variants' first requests from colliding in the on-disk response cache. That detail matters later.
Reading the steps
The results JSON keeps each task's final answer and call count, not the steps. The response cache from the July run was still on my disk, though (it's gitignored). I replayed both models through the benchmark code with a provider that raises on any cache miss. That made 0 network calls, and all 60 ReAct rows (15 tasks, 2 variants, 2 models) came back identical to the committed results. So I could read every step the agents took.
Where the one task went
When the verifier accepts, the verify loop behaves exactly like the plain one. With zero rejections, the only thing different between the two Gemini runs was that sentence in the prompt. The searches drifted apart anyway. The verify variant lost t03 and t14, both about Lake Azurine, and won t02, which the plain loop lost. Net result: one task down.
On t14 the verify agent searched for "Lake Azurine", "Lake Azure", "What is Lake Azurine?", "\"Lake Azurine\"", "Lake Azurine location" and "What country is Lake Azurine in?". Six searches, six "No results found", budget gone. The plain agent tried plain "Azurine" on its second step and got the fact.
The search tool was the real problem
Here is the matching logic:
key = query.strip().lower() if key in _FACTS: return _FACTS[key] for fact_key, fact_value in _FACTS.items(): if key and (key in fact_key or fact_key in key): return fact_value return f"No results found for '{query}'." The fact is stored under "azurine lake". "lake azurine" isn't a substring of that key, and the key isn't a substring of it. "capital of Ostrenia" misses "ostrenia capital" the same way, while bare "Ostrenia" hits because it sits inside the key. The tool rewards short, oddly ordered queries and punishes the way people (and models) phrase things.
Across both Gemini variants the model made 125 search calls and no lookup calls; the one worked example in the prompt only uses search. 64 of the 125 searches came back empty. Each of Gemini's 8 budget stops spent at least 3 of its 6 steps on empty searches. In 3 of them the fact the agent needed arrived on step 6, with no step left to answer.
So 10/15 against 9/15 measures how a model wanders through a brittle tool under two slightly different prompts. It tells me nothing about verification.
On Haiku the verifier blocked right answers
The same code against Claude Haiku 4.5 scored 5/15 plain and 1/15 with the gate. Here the verifier was busy: 14 calls, 10 rejections. Half of those rejections hit answers the grader counts as right (Marveth once, drenn twice, Vessic twice).
Haiku often wrote out a whole imagined trajectory in one reply, observations included. 96 of its 105 search arguments contain the text "Observation:". My parser:
_ACTION_RE = re.compile(r"Action:\s*(\w+)\[(.*)\]", re.DOTALL) The greedy .* with DOTALL runs to the last ] in the reply, so the invented text became the search argument and stayed in the trajectory. On t01 in the verify run, Haiku's first search argument starts like this:
Kelvorin River] Observation: The Kelvorin River is a river in Scotland. The real tool replied that the river flows through Ostrenia, the agent went on to find Marveth, and the verifier rejected Marveth. Its reason: "The trajectory contains a critical inconsistency". It had read "Scotland" and "Ostrenia" as two observations that contradict each other. Given what it was shown, it had a point. My parser had polluted the trajectory it was judging.
The loose matcher makes it worse. A long argument full of invented text still contains a fact key like "sablewood forest", so search keeps handing back the first-hop fact while the agent asks for the capital.
The one direct success is a grading bug
Answering directly with no tools scored 1/15 on both models, and it was the same task each time:
("t12", "Which country's capital is Northarrow, given it contains Cindrapass?", "Northarrow", ("cindrapass", "grendale capital")), The question asks for a country, and the gold answer is the capital. Gemini's direct reply began "There is no country with a capital named Northarrow" and passed the substring check. All four ReAct runs on t12 answered "Grendale", which is right, and were scored wrong. The true direct score is 0/15, and every ReAct number above is one task low. I have since fixed the task and regraded the stored answers, so the repo now reports 11/15 and 10/15 for Gemini and 6/15 and 2/15 for Haiku, with the first grading kept in a table beside them. The numbers in this post are the ones I first published.
Limits
- N is 15, with one run per variant per model. A one-task gap is noise until it repeats.
- I can't separate the prompt sentence from ordinary drift. The results file itself notes that the ReAct loop "is not perfectly deterministic at temperature 0". Either way the cause isn't verification, and that is the only claim I'm making about the gap.
- The step-level claims come from a cache that lives on my machine. The committed results agree row for row, but a reader can't check the steps without rerunning.
- The verifier is the same model as the agent. A separate verifier model might behave differently.
- "Right" here means the benchmark's lenient substring grader, t12 bug included.
- I haven't rerun anything with a fixed tool or parser, so I don't know what the scores would be.
What I would do differently
Read the steps before writing the explanation. My results paragraph told a reasonable story that one look at the steps would have killed.
Save steps in the results file. I got lucky that the cache survived.
Report the verifier as its own metric: how often it rejects, and how often a rejected answer was right. 0 of 11 says the gate never acted. 5 of 10 says it is blocking good answers.
Match tool keys by words. I checked a version that returns a fact when every word of its key appears in the query. It would have answered 49 of Gemini's 64 empty searches, each with the right fact:
import re from benchmarks.bench_react import _FACTS def words(text): return set(re.findall(r"[a-z]+", text.lower())) def search_by_words(query): """Return the first fact whose key words all appear in the query.""" q = words(query) for key, fact in _FACTS.items(): if words(key) <= q: return fact return None Stop parsing the action at the first ], so a model that writes its own observations can't smuggle them into the trajectory.
Let both variants share the cache. I changed the prompt to avoid collisions, but a collision is what I wanted: with identical prompts, identical requests get identical responses, and the verify run would split from the plain run only at a rejection. That is the comparison the benchmark was meant to make.
The results files record $0.0097 for the Gemini ReAct run and $0.2576 for Haiku's. The code and per-task results are in github.com/SarthakB11/agentic-patterns, in benchmarks/bench_react.py, patterns/react/verify.py and benchmarks/results/.
