How a Silent Fallback Corrupted My Hindsight Experiment

How a Silent Fallback Corrupted My Hindsight Experiment
Our first comparison of an agent with and without memory looked promising. By the third session, the agent using Hindsight memory had zero overfit or error runs. Then we looked at a column we had logged almost by accident, and the result fell apart.
This is the story of that bug, how we found it, and what we changed.
The setup
Ninaivu is an ML lab notebook agent. It trains real scikit-learn models on the Telco Customer Churn dataset from Kaggle, remembers every experiment, and uses that history to propose the next one. The code is public at github.com/divya240707/Ninaivu. The memory layer is the open-source agent memory system Hindsight, and the Hindsight documentation explains the operations we used. For the idea in general terms, see what agent memory is.
To test whether memory helps, we ran two agents with the same model and prompt, 3 sessions of 10 runs per trial, with the context wiped between sessions. Only the memory agent could recall earlier experiments from Hindsight.
The bug
The agent asks a language model for the next configuration. If the call failed twice, the code didn't stop. It quietly fell back to a random configuration, and returned a flag saying so:
return random_config(), True
That flag went into a column called fallback in our results file. We added it as a bookkeeping detail. We did not think we would need it.
The result that looked good
After three trials per agent, the chart showed the memory agent dropping to zero overfit or error runs in session 3, while the baseline kept making them. We kept going to six trials to make the result firmer.

It wasn't firmer. With six trials per agent, the picture was this (a bad run is an overfit or error run, out of 60 per cell):

Mean test F1 was also not better for memory in session 3 (0.545 against 0.574).
Looking at the fallback column
The table's last two columns were the surprise. In 64 of 360 runs, the agent hadn't made a decision at all. In session 3, 19 of the memory agent's 60 runs, nearly a third, were random configurations. Random deep forests tend to overfit, so a pile of overfit runs didn't mean the memory agent was learning badly. It meant it often wasn't being asked.
The early "zero bad runs" result came from a small number of trials, with random fallbacks mixed in. The comparison wasn't measuring memory, and neither could we say what it was measuring.
Finding the cause
We had no record of why the calls had failed. We changed the code to print the exception, then reran. This is what came back:
RateLimitError: Error code: 429 - Rate limit reached for model openai/gpt-oss-120b ... tokens per day (TPD): Limit 200000, Used 199537, Requested 1093.
The free tier's daily token limit was nearly used up, and each request cost about 1,100 tokens. That is the most likely cause of the earlier fallbacks too. We never logged the cause for those runs, so we can say it is likely, not proven.

What we changed
Made rate limits stop the run. A failure the run can't recover from should not turn into data:
except Exception as exc:
if type(exc).name == "RateLimitError":
raise SystemExit(f"rate limited: {str(exc)[:300]}")
print(f"proposal failed: {type(exc).name}: {str(exc)[:200]}")
time.sleep(2)
Kept the contaminated data, labeled. We renamed the files so they stay in the repository for reference, and wrote in the README that they were affected.
Switched to a smaller model (openai/gpt-oss-20b), which has its own token allowance.
Reran from scratch and checked the fallbacks column before reading anything else. It was 0 in every row.
Because the model changed, the two sets of numbers are not comparable, and we don't compare them.
What the clean data said
With 2 trials per agent and 0 fallbacks, mean test F1 was 0.573 for the baseline and 0.574 for the memory agent. In sessions 2 and 3, the memory agent made 4 bad runs out of 40 and the baseline made 9 out of 40. In session 1, where neither agent has memory, the counts were 4 and 5. With so few trials, we call that a pattern and not a result.
The exciting "zero bad runs" chart is gone. The honest one is smaller, and we can defend every number in it.
Lessons
Fallbacks should be loud. A silent fallback turns an infrastructure problem into fake behavior. Count it, print it, and stop on the ones you can't recover from.
Log provenance for every data point. Our fallback column was the only reason we caught this.
Be suspicious of the result you wanted. It deserves more checking, not less.
Read free-tier limits before you plan a run. About 1,100 tokens per request adds up fast across hundreds of calls.
Keep discarded data, labeled. It shows your work and what you ruled out.
The code, data and discarded runs are in the Ninaivu repository. To try agent memory yourself, start with the Hindsight documentation.
#AIAgents #AgentMemory #Hindsight #LLM