Agents That Forecast Well Can Still Lose Money: The Lesson for Small Businesses Handing Over Transactions

The most useful AI experiment this month was not a model launch. It was a research lab handing nine frontier agents a sandbox, a week, and a $500 data budget, then watching what happened when other agents tried to make real money from their answers.

On the November 7 episode of The Cognitive Revolution, Brian Zhou of Thoughtful walked through BallotBench. Nine agents — described as the best model each major lab offers — each worked alone in its own sandbox for seven days, forecasting winners, odds, 80% confidence ranges, and margins for roughly 800 federal and statewide races. The headline result: in Zhou's words, the models "perform just as well as the best forecasters," slightly behind the markets in the House, and in the Senate behind forecaster Nate Silver but ahead of the markets.

Then came the part that should matter to you.

Thoughtful also ran a trading layer: one forecaster agent and one trader agent per team, with $50 of real money on the prediction market Kalshi plus $10,000 simulated. The trading agents did quite well the week before election day. On election day itself, they lost. Zhou's diagnosis, from the traces: the forecaster agents saw a result and passed it along, but by the time the trader acted, "more programmatic or more deterministic algorithms and bots have already traded on this information" — and sometimes the information passed between agents was wrong. "Models making up information, parsing information wrong... partly explains why traders didn't do too great."

Two agents. Both competent at their own job. The forecast quality held up — and the money still left.

The forecasting skill and the transaction skill are different skills

The lesson is not "AI can't predict elections" — the forecasting results were strong, and BallotBench's own scoring waits on certified results, so treat the placement claims as preliminary. The lesson is that forecasting well and transacting on a forecast are two different competencies, and the failure mode lives in the seam between them.

For a small business, this is exactly the shape of most things you would hand to an agent. "Should we take this client?" is a judgment call. "Email them a yes and book the kickoff" is a transaction. "This invoice looks late" is a judgment call. "Send the reminder" is a transaction. The experiment above suggests the second one is the harder thing to automate safely — not because agents can't write the email, but because the cost of a wrong action is asymmetric. A bad forecast you can ignore costs nothing. A bad transaction you executed costs money, a customer, or your reputation.

Where agents actually break: three failure modes from the traces

The BallotBench traces showed three concrete ways a competent-looking agent fails under real stakes. Watch for these in your own setup:

1. Speed doesn't transfer. The trading agents lost on election day because markets moved faster than the agent loop could react. In your business, the equivalent is any agent acting on time-sensitive information: a price change, a stockout, an expiring discount. If your agent checks email hourly and your supplier's pricing updates at 9:02 AM, the agent's action at 9:45 AM is based on stale reality.

2. Information degrades between agents. The single biggest finding was "models making up information" in the handoff from forecaster to trader. In your business, the equivalent is chained automations: an agent reads a customer email, summarizes it for a second agent, which drafts a reply. Each hop is a chance to flatten a nuance or invent a detail. The customer asked about the December delivery; somewhere in the chain it became "delivery."

3. A strong forecast invites overtrust. Zhou's results were genuinely good — and that's the trap. A week of good trades makes election-day losses feel like bad luck rather than a structural limit. In your business: an agent that handled 30 invoices cleanly does not mean the 31st is safe, and the failure will come precisely when conditions change (a new customer type, a holiday, a vendor outage) — the moment that looks most like "election day."

The rule: separate the yes/no from the transaction

Here is a practical pattern for any small business experimenting with agent-handled transactions, translated from what went wrong in the experiment:

• Keep the judgment call and the money-moving action as separate steps, with a human owning the seam. Let an agent produce the forecast — "this quote looks profitable," "this refund claim seems valid" — but keep a human pressing send on anything that moves money, touches a real customer, or is time-sensitive. The experiment's failure was specifically the seam; make the seam a person.
• Never chain two agents where one will do. The forecaster-to-trader handoff is where the invented facts appeared. If your workflow needs research plus action, have one agent do both and show its work, rather than agent A summarizing for agent B.
• Time-box time-sensitive actions. If an agent acts on anything that decays — pricing, inventory, market data — define how stale an input can be before the agent must stop and ask. The trading agents had no such rule; that's why "didn't react fast enough" turned into losses.
• Define the asymmetric-loss boundary in writing. List which actions are cheap to get wrong (internal drafts, research summaries) and which are expensive (outbound customer communication, payments, contract changes). Agents get free rein on the first list. The second list requires your click, every time, until you have run enough clean passes on changed conditions to trust it.

The honest boundaries

• One election is a small sample, and BallotBench's own scoring waits on certified results — the "as well as the best forecasters" placement is Zhou's preliminary read, not a final published ranking. The trading losses, though, are directly observed in the traces, and that asymmetry is the point.
• The trading agents ran with tiny stakes ($50 real per agent). It is an open question how failure modes change with real volume; do not read this as "agents can't trade," only as "competence in one step does not transfer to the next."
• The agents ran in ready-made agent tools that BallotBench did not build, so some behavior is the tool's harness, not the model. That cuts both ways: your agent tool's harness will have its own quirks you can't see.

The structural point survives all the caveats: forecasting skill is real, transaction skill is separate, and the seam between them is where the money went. For a small business, the safe version of an agent economy is not "automate the whole transaction" — it is "let the agent earn the forecast, and keep the send button."

If you want the systems side of this — which workflows are worth handing to agents in the first place, and how to wire them so review stays cheap — the AI Operations Playbook covers agent blueprints for support, content, sales, research, finance, and project management, with review checkpoints built into each one.