Stop dividing by 4. I counted 951 texts and the rule is wrong both ways.

Most token guides say one token is about four characters, and a lot of code runs Math.ceil(text.length / 4) right before an API call without anyone checking it.

I build an LLM cost calculator, so this number ends up inside every estimate I make. This week I finally counted. I ran 951 text samples through OpenAI's o200k_base tokenizer (GPT-4o and newer) with tiktoken 0.14.0.

Characters per token

For plain English the rule guesses too high, so it errs on the safe side. For JSON, CSV, IDs and emoji it guesses about half. If you're building agents or RAG, the second group is most of what goes into your prompts: tool results, API responses, exported tables. That's the error that shows up on the bill.

Why JSON comes out so badly

Tokenizers learn mostly from prose, so a common English word and the space before it are often a single token. Numbers get split up. 4821.37 or 2026-07-14 turns into several short pieces, every quote, colon, comma and brace counts, and JSON repeats every key on every record.

Random strings are worse still. An API key or a SHA hash has nothing the tokenizer has seen before, so it comes out at about 1.5 characters per token.

The averages hide a lot of spread, too. My JSON files went from 1.7 to 2.8 characters per token depending on how many numbers they had in them, and CSV from 1.6 to 3.9.

What it does to a budget

Take an agent that reads 40,000 characters of JSON tool output per call, 10,000 calls a month.

• length / 4 says 100 million input tokens a month.
• At 2.11 characters per token it's about 190 million.

At Claude Sonnet 5.5's input price ($2 per million tokens) you'd budget $200 and pay about $379, before any output tokens.

Context limits go wrong the same way. A payload you think is 100K tokens is closer to 190K, so it doesn't fit in a 128K window anymore. Usually you find out from a truncation error, or from an agent that has quietly dropped the start of its instructions.

I built a better estimator, and it missed too

The fix I tried first: stop using one divisor and count words, digits, punctuation and CJK characters separately, each at its own rate. I fitted the rates on the 951 samples and got a 26-line JavaScript function that was within about 2% on every type.

That 2% was measured on the same samples I fitted it to, so it didn't prove much. To get a fair test I took 832 real files none of the estimators had seen, the first 4,000 characters of README, package.json and index.js files from npm packages, and ran mine, length / 4 and the tokenx library on them.

1.00 is exact. 0.89 means the estimate counted 89% of the tokens you'd actually be billed for.

Mine undercounted real package.json files by 11.5%. All my JSON training samples were tables of numbers. Real package.json files are full of version strings like ^7.21.0, @scoped/package-names and two-space indentation, and my samples had none of that. tokenx, which did worse than mine on my own samples, was within 1.4% here.

tokenx has a weak spot as well. On my samples it counted only 38% of the tokens in API keys and hashes and 43% in messages with a lot of emoji.

So I don't think any estimator is safe in general. Each one is about as good as the text it was tested on, and most of us never test it on our own text.

What I'd use

• On a server, or anywhere a wrong count breaks a request or a budget, run the real tokenizer. gpt-tokenizer and js-tiktoken both matched Python tiktoken exactly on all 951 samples. The usage numbers the provider returns work too.
• In the browser it's a size question. The exact tokenizers are about 1.1 MB gzipped because the vocabulary has 200,000 entries. tokenx is 1.2 kB and mine is 0.5 kB. For a live counter on English, code or Markdown, either is fine if you label the number as approximate.
• If the text has keys, hashes, UUIDs, base64 or emoji, I wouldn't rely on tokenx alone.
• For real-world JSON, leave at least 15% headroom whichever estimator you use.
• I'd stop using length / 4 for JSON, CSV or Chinese. It missed 29% to 65% of the tokens on each of them.

If you want something you can do in your head: divide characters by 5 for plain English, 4.2 for code, 2 for JSON, CSV and logs, 1.5 for keys and hashes, and multiply Chinese or Japanese characters by 0.73.

Limits of this test

• It only covers OpenAI's o200k_base. Claude, Gemini, Llama and DeepSeek use their own tokenizers, and the same text can come out at a different count. If the number matters, use the provider's token counting endpoint.
• My CSV and JSON samples are generated tables, and the unseen files all come from one project's npm dependencies, so they're mostly English and code. Business documents or chat logs could rank the estimators differently.
• The Chinese set is 11 paragraphs, and Japanese, Korean and Russian are four sentences each. Don't read too much into those rows.

Try it on your own text

The samples, the counting script, the comparison script and the estimator are all in aicostbudget/tokenizer-ratios (code MIT, data CC BY 4.0).

git clone https://github.com/aicostbudget/tokenizer-ratios cd tokenizer-ratios npm install npm run compare 

To check your own data, add a few lines to data/samples.jsonl (one JSON object per line with a category and a text) and run it again. If you find a kind of text where all three estimators are badly off, please open an issue with a sample and I'll add it.

The longer versions, with charts, are on my site: how many characters a token really is and which token estimator to trust.