Study Finds Unequal Token Costs Across Languages in AI Models
publishers 109articles 347first reported 23 Sep, 14:00 UTCdeveloping since 30 Sep · 32 editions
People using artificial intelligence in less widely represented languages can face higher costs and slower performance because models require more tokens to process their text, according to a study published by the Coalition for Language Equity in AI Research.
The researchers described the disparity as a hidden “tokenizer tax.” Among the 20 most widely used languages online, the most expensive required between 1.5 and five times as many tokens as the least expensive. When the analysis expanded to 200 languages, the gap reached 17-fold in one model.
Higher token counts can increase usage fees and latency while reducing the effective space available for documents and conversation history. Pratyush Choudhury, co-founder of the India-focused AI investment fund Activate AI, said this can affect service quality when applications need to retain earlier interactions, such as a tutoring system remembering a student’s mistakes and explanations.
The study found that a language’s rarity was not the only factor. Script representation in model training also played a major role. Asturian benefited from using the Latin script and its relationship to better-represented Romance languages, while Burmese, Tibetan, Tifinagh, Khmer, Lao and Ol Chiki were consistently among the most disadvantaged scripts across the models analyzed.
Raoul Nanavati, chief executive of Navana.ai, said open-data projects could improve language processing, citing an initiative that collected native-speaker text and speech from 38 dialects.
HOW THIS STORY WAS MADE
Written from 8 articles, 6 with the publisher's own text; 8 independent newsrooms once syndicated copies count as one; this version written 642 h after the record first saw the story; the editor kept it as written.
- Publishers
- 109
- Source articles
- 347
- Given to the writer
- 8, 6 with the article's own text
- Independent newsrooms
- 8
- This version written
- 642 h after the record first saw the story
- Second model (editor)
- kept as written
- Publication gate
- passed
- Human review
- none
WHAT THE EDITOR DID
The second model checked the writer's draft against the same sources and published it as written.
Written by a language model from the sources above, then checked by a second model that may only cut, attribute or correct. How it works →