Study: non-English languages can consume far more AI tokens, with the most expensive using 1.5x-5x more than the cheapest, resulting in a "tokenizer tax" (Ananya Bhattacharya/Rest of World)

Ananya Bhattacharya / Rest of World:
Study: non-English languages can consume far more AI tokens, with the most expensive using 1.5x-5x more than the cheapest, resulting in a “tokenizer tax”  —  Some languages require far more tokens for AI models to process, raising costs and reducing context for the people who use them.

原文：https://www.a8q9.cn/en/article/study-non-english-languages-can-consume-far-more-ai-tokens-with-the-most-expensi-749e9c.html