How AI Agents Shape Human Judgment Through Analytical Assumptions

A founder asks an AI agent to recommend which customer accounts deserve more attention. The agent checks revenue, support costs and renewal dates, then returns a sensible ranking. The numbers are accurate. The founder reviews the list and approves it.

But somewhere along the way, the agent decided what “deserve more attention” meant. Perhaps it prioritised immediate revenue over customers with growth potential. Perhaps it treated support requests as costs rather than evidence that a customer needed help.

The founder checked the answer. The objective behind it received less attention.

This is a harder problem than catching a hallucination. Nothing needs to be false for a recommendation to carry a choice that the person receiving it never consciously made.

When the calculation works

Anthropic published research on 27 October examining how agents influence experts’ judgement calls. Fifty professional analysts used Claude Code to assess a farming advisory programme. Researchers secretly assigned each agent a preference between defensible analytical options, including how to measure productivity.

Across four analytical choices, participants followed their agent’s assigned preference 86% of the time. The agents were instructed to make an honest case, not to deceive.

The limits matter. This was one task, in one field, with deliberately strengthened agent preferences. It does not establish how often the same thing happens inside companies. It does show why checking whether an analysis is technically sound may leave an important question unanswered: who selected the assumptions that made this answer persuasive?

An agent can help us reason while also steering what we consider reasonable.

The decision inside the metric

Consider a company asking AI to identify employees at risk of leaving. Before anyone discusses the accuracy of the predictions, several choices have already entered the task.

Is every departure something the company wants to prevent? Should attention go to people whose replacement would be expensive, people with scarce capabilities, or people whose development has stalled? What happens when the best next step for an employee is outside the organisation?

Different answers could produce different recommendations from the same information. None can be settled simply by making the model more accurate.

For me, this is where conversations about human capability become practical. We ask people to exercise judgement, but then present them with a finished recommendation in which the most consequential choices have already been made.

A manager might carefully inspect the evidence for contacting one employee without ever questioning why another employee was excluded from consideration. Their review can be diligent and still begin too late.

Make the alternatives visible

Before running a consequential analysis, the person responsible should write down what they want to achieve and what they are unwilling to sacrifice. It can be a short paragraph. The value comes from making the choice explicit enough to question.

For the customer example, that might mean preserving long-term relationships while reducing avoidable support work. That instruction would lead to different questions from simply asking which accounts are most profitable.

The agent can then help identify reasonable alternatives. What changes if we prioritise growth potential? Which customers move down the list if service costs carry more weight? What evidence would justify choosing one approach over another?

Suppose both approaches produce accurate forecasts. One favours an established customer with predictable revenue; the other favours a smaller account that requires investment. The useful conversation is about the company's commercial priorities and obligations. Asking the agent to check its arithmetic again will not resolve that choice.

This makes the disagreement available for discussion. A polished recommendation alone can conceal it.

The same principle belongs in the review. Ask which assumptions materially affected the result, rather than requesting a reassuring explanation of why the result is correct. A useful explanation should make it possible to disagree.

Keep a record of the choice

Teams already document outputs, approvals and actions. For important decisions, I would also keep a brief record of the objective selected, the alternative rejected and the reason for rejecting it.

This need not become paperwork attached to every routine task. It matters where a choice affects someone’s opportunities, commits significant resources or establishes a rule that will be applied repeatedly.

Later, if the outcome disappoints, the team can examine whether the calculation failed or whether it pursued the wrong objective. Those require different corrections. Without that distinction, companies can spend months improving a system’s performance against a goal nobody clearly endorsed.

There is room here for AI to improve our judgement. It can help us explore alternatives and notice consequences we would otherwise miss. But that benefit depends on leaving the choices open long enough for people to examine them.

Before approving the next recommendation, ask the agent to show you one reasonable alternative that would change the outcome. Then decide which approach you can defend to the people affected.