CLEAR benchmark measures up to 12x tokenizer cost disparity across languages Other

The Coalition for Language Equity in AI Research (CLEAR) published an empirical audit revealing that non-English languages face inference cost premiums of up to 12x due to tokenizer bias. The benchmark, dataset, and analysis tools are open-sourced on GitHub and Hugging Face.

Most state-of-the-art LLMs deploy subword tokenizers trained predominantly on English corpora. Using the new TokenCostEquity benchmark across 15 foundation models (including GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and Llama 3) on the FLORES evaluation set, CLEAR quantified the token inflation ratio relative to English.

While Western European languages incur a 1.3x to 1.8x token penalty, non-Latin scripts such as Cyrillic, Arabic, and Devanagari require 2.5x to 4x more tokens for equivalent text. Low-resource and African languages incur severe overheads, requiring 8x to 12x more tokens per sentence, multiplying both API pricing and inference latency for non-English users.

What it is

• TokenCostEquity benchmark. Evaluates token fragmentation ratios across 100 languages and 15 commercial frontier models.
• Script disparity. Non-Latin scripts (Russian, Arabic, Hindi) generate 2.5x to 4x more tokens than English for identical semantic content.
• Extreme penalties. Select low-resource languages require up to 12x the token budget of English.
• Latency impact. Higher token counts directly proportional increase time-to-first-token and overall generation latency.
• Open research. CLEAR published the evaluation harness on GitHub and benchmark data on Hugging Face.

Sources

• CLEAR: The Tokenizer Tax
• GitHub: CLEAR Token Cost Equity repository