Our token classifier gave us a 10/10 grade from four tasks we typed in ourselves

2026-11-16来源 dev.to AIEnglish

We have a small script that sorts coding-agent token spend into three buckets: teach (reusable skills and context), produce (output that shipped) and spin (retries and wandering). I ran it and it printed "GRADE A+ (10/10)". Then I read the source, and the grade had nothing to do with our usage.

What was wrong

The script did not read any usage. It had four sample tasks written into the file, each with a token count and a cost we had typed in by hand. It sorted those four, noticed that spin was under 10 percent, and printed an A+. The grade line was a string in the code, so no input could ever change it.

Two smaller defaults made it worse. A record with no cost silently got costUsd = 0.001, so a missing number turned into a plausible-looking number. A record with no acceptedTask flag counted as accepted. Both defaults pushed the result toward "looks fine".

Nothing here was a crash. The output looked like a measurement, and the only way to catch it was to ask where the numbers came from.

The fix

The sample data and the grade are gone. The tool now takes measured records from a file or stdin:

node tools/token-spend-classifier.js --input usage.json --max-tokens 100000 --json 
  • No records, or a record without inputTokens and outputTokens: exit 1.
  • Cost not measured: costUsd stays null, and the total cost and cost per accepted task stay null too, so the report says "unknown" instead of guessing.
  • Category not given: UNKNOWN. Accepted not given: not accepted.
  • --max-tokens N makes it usable as a gate in a script: exit 0 while measured tokens are under N, exit 2 once they reach it. It only checks usage you hand it. It does not reserve tokens or stop a call that is already running.

The new test feeds it empty input, malformed JSON, a negative token count and a non-numeric budget, and checks each one fails closed. It also checks that nothing in the output contains the old grade string.

What to check on your own machine

If you have a script that reports agent spend, look for three things:

  1. Where does each number come from? If you can't point to a log or an API response, it is a sample.
  2. What happens when a field is missing? A default that makes the result look better is a bug. Prefer null, and print "unknown".
  3. Can the output ever be bad? A health score that can only say "fine" is decoration. Give it a path to fail, then test that path.

Our own copy of this tool printed a perfect grade whatever the input. The grade was never wrong in a way a test would catch, because the test asserted the A+.

The guard we are building for Mac agents is open source: https://github.com/IgorGanapolsky/mac-yolo-safeguards

原文:https://www.a8q9.cn/en/article/our-token-classifier-gave-us-a-10-10-grade-from-four-tasks-we-typed-in-ourselves-5bf438.html

相关文章