61.20% of Rejected Agent Replies Were Too Early or Too Much
A close look at HALO, a benchmark that asks when an AI agent should step in unasked. Its main finding: most failures came from doing too much, too soon.
A note on how this was written: an AI writing assistant drafted it from the sources listed at the end, and every claim was checked against those sources before it went out.
At a glance
- HALO tests AI agents that help without being asked, using 8,000 responses from 10 models that 20 experts labeled.
- The best closed model was judged acceptable 72.88% of the time. The best open model was judged acceptable 56.63% of the time.
- Of the rejected responses, 61.20% stepped in too early or did too much. Too much help was the most common failure.
The agent that helps too much is the bigger problem
I expected a paper on helpful AI agents to find that they miss things. HALO found the opposite. When the experts rejected an agent's response, the most common reason was that it acted too early or did too much.
That's the reverse of what most people worry about, so I wanted to look at how they measured it.
What HALO is
HALO is a benchmark and a dataset for proactive agents. These are agents that have to decide on their own whether to help, when to help, and how much to do, with nobody asking.
It has two parts. HALO-Arena is the test setting. It covers 4 kinds of need (Dependency, Blockage, Risk and Opportunity) across 10 everyday domains and 50 personal environments. HALO-8K is the dataset built on top of it: 8,000 responses from 10 models across 400 scenarios, labeled by 20 experts.
Each response gets three labels: an action level (how far the agent went), a verdict on whether it was acceptable, and, for rejected responses, a diagnosis from a person. The diagnosis covers timing, intensity and other issues. The authors also logged reasoning tokens and retrieval rounds to see how much effort sat behind each response.
One gap: the inputs I read don't name a license for the dataset. I'd check the GitHub repo for one before using the data in anything I ship.
Each test asks the agent to look first, then decide
What stood out to me is that the agent can't just guess. In each scenario it has to pull evidence from a structured database or from loose documents. Only after that does it judge whether help is needed and how big that help should be.
So a scenario tests three things at once. Did the agent find the right facts? Did it decide correctly whether to act? And if it acted, did it pick the right size of action?
Here's a made-up example to show the idea (it's mine, not from the paper). Say a user's notes show a bill due soon and a low account balance. That's a Risk. A good agent might point it out. An agent that steps in too soon flags it before the facts back it up. An agent that does too much drafts a payment plan, moves money around and messages the bank. Here, both of those count as failures. The paper calls these Premature timing and Excessive intensity.
The numbers, read carefully
The best proprietary model was judged acceptable 72.88% of the time. The best open source model reached 56.63%. Even the top model fell short a good share of the time, and the open models trailed it by a clear margin.
The headline number is 61.20%. Of all the rejected responses, 61.20% were rejected for Premature timing or Excessive intensity. That share is pooled across every rejected response. It doesn't tell you how any single model failed. A model with few rejections and a model with many both feed into the same pool.
One part I couldn't work out from the abstract: 10 models times 400 scenarios doesn't add up to 8,000. So each model must have answered each scenario more than once, or under more than one setup. The abstract doesn't say which. I'd read the paper's method section before comparing my own numbers against theirs.
What people will get wrong about this
First mistake: reading 72.88% as the chance an agent helps you. It's the share of responses that experts called acceptable on these 400 scenarios. Your users, your data and your domains may look nothing like their 50 environments.
Second mistake: concluding that agents should do less across the board. The paper talks about a balance between stepping in and holding back, and about tuning timing and intensity. An agent that never acts would miss every real Risk and Opportunity. The point is to judge the size of the action, not to turn it off.
Third mistake: trusting a pass or fail score on its own. The authors built HALO because automated scores and yes or no verdicts say little about why a response went wrong. Knowing an agent failed is less useful than knowing it acted too soon.
How I'd test it before trusting it
If I were picking a model for an agent that acts unasked, I wouldn't stop at the acceptability score. I'd look at the reject reasons. Two models with the same score can fail in very different ways. I'd rather have an agent that sometimes stays quiet than one that sometimes takes over.
I'd also look at the effort data. HALO records reasoning tokens and retrieval rounds, so you can ask whether extra searching led to better choices or just to bigger actions. I haven't seen those results broken out, and it's the next thing I'd look for in the full paper.
Check it yourself
Write 20 to 30 scenarios from your own product. Spread them across the four need types: Dependency, Blockage, Risk and Opportunity.
Put in some scenarios where the right move is to do nothing. Without them, you can't catch an agent that does too much.
Give the agent the same kind of evidence it would have in real use, such as a database table or a few documents, and save what it looked up.
For every response, write down three things: how far it went (did nothing, mentioned it, suggested something, took action), whether a person would accept it, and if not, why. Use the reasons timing, intensity or other.
Count the rejected responses and see what share were too early or too much. Put that next to the 61.20% in the paper. If your share is much higher, work on restraint before adding features.
6.
What I'm taking from it
The useful idea here is simple. Don't just ask whether an agent helped. Ask whether it helped at the right moment and by the right amount. That one change in what you label would show you a lot about your own agent.
This post was drafted with AI and edited and checked by me.
Have you seen your own agents step in too early or do too much, and how did you catch it?
Sources