Equivalent meaning can enter the model with unequal representation cost.
Multilingual evaluation usually starts with model output. The Tokenization Equity Audit (TEA) moves one step upstream and audits the token sequence produced before inference. Prior work has shown that cross-language tokenization disparities can affect cost, latency, and usable context [1][2].
The layer TEA audits
Why this matters for underserved-language educational AI
API systems: token-based billing makes longer encodings more expensive.
Context: nominal windows are measured in tokens, not semantic content.
Local deployment: avoiding an API bill does not remove sequence-length overhead; longer sequences still cost local compute and memory.
One Python error message: 8 tokens in English, 22 in Tamil.
Corpus item T1-10 — “IndexError: list index out of range” — encoded using GPT-4o
o200k_base. No model has run yet; every difference below is produced entirely by the
encoder vocabulary.
A custom programming corpus preserves how multilingual technical teaching actually works.
TEA uses beginner Python because technical tutoring naturally mixes native-language explanation with English error names, identifiers, and code — a poor fit for generic parallel prose.
120 parallel items · six languages
English source items were translated into Bengali, Hindi, Arabic, Tamil, and Yoruba. Python
identifiers and error names such as TypeError, IndexError,
NoneType, and code variables are intentionally retained in English, matching real
classroom code-switching.
Tokenizers and validation scope
Tokenizers audited: GPT-4o o200k_base, Qwen2.5-7B, and Mistral-7B-v0.1.
The audit uses tokenizer vocabularies only — no model inference.
Bengali items were reviewed by two Bengali speakers with programming experience; Hindi items by a Hindi-speaking contributor familiar with technical usage. The custom corpus also enables a Bengali code-switching sensitivity check: “clean” items contain at least 75% Bengali-script alphabetic characters.
Three quantities carry the whole audit.
Every result on this page is derived from these three definitions.
tokens(language) ÷ tokens(English)
Token Fertility Ratio — token premium relative to English, computed per item under the same tokenizer.
nominal window ÷ TFR
Effective Context Window — English-equivalent semantic content that fits inside a fixed nominal token budget.
price × (TFR − 1) × mean English tokens × requests
Illustrative API cost premium. Dollar values move with provider pricing; the TFR ratio is the durable result.
Every tested non-English language is above English parity under every tokenizer.
Mean Token Fertility Ratio (TFR) across all 120 items. English is fixed at 1.00× under every tokenizer. Click a legend swatch to isolate a tokenizer.
View full results table (Table 1, mean ± SD)
| Language | GPT-4o | Qwen2.5-7B | Mistral-7B |
|---|---|---|---|
| English | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 |
| Arabic | 1.44 ± 0.21 | 1.70 ± 0.27 | 3.86 ± 0.54 |
| Hindi | 1.72 ± 0.31 | 4.86 ± 0.91 | 5.20 ± 0.99 |
| Bengali | 1.56 ± 0.41 | 4.50 ± 1.46 | 4.44 ± 1.45 |
| Tamil | 2.09 ± 0.48 | 6.55 ± 1.80 | 6.57 ± 1.82 |
| Yoruba | 2.37 ± 0.49 | 3.18 ± 0.63 | 3.33 ± 0.67 |
Tokenizer family matters
Bengali: 1.56× → 4.50× / 4.44×
GPT-4o versus Qwen2.5-7B and Mistral-7B-v0.1. The same language carries very different sequence overhead depending on tokenizer design.
Script family is not enough
Yoruba 2.37× > Bengali 1.56×
Under GPT-4o, Latin-script Yoruba is more expensive than Bengali — evidence for measuring language-level behavior rather than assuming fairness from script alone.
Open-weight trade-off
Tamil exceeds 6.5×
Qwen2.5 and Mistral remove commercial API billing, but their tokenizers carry much longer sequences for several languages in this corpus.
English code-switching suppresses the Bengali premium rather than inflating it.
Choose a corpus tier to compare all Bengali items with the “clean” Bengali subset (at least 75% Bengali-script alphabetic characters). The clean subset is more expensive in every tier.
View full sensitivity table (Table 3)
| Subset | Tier 1 | Tier 2 | Tier 3 |
|---|---|---|---|
| All items (n=120) | 1.65 ± 0.52 | 1.55 ± 0.36 | 1.50 ± 0.36 |
| Clean only (n=62) | 2.11 ± 0.57 | 1.79 ± 0.17 | 1.79 ± 0.10 |
A nominal 128k-token context window does not carry equal semantic capacity across languages.
Select a language. TEA estimates effective English-equivalent content by dividing the nominal context window by mean GPT-4o TFR.
Bengali retains about 64% of the English-equivalent capacity in a nominal 128k window.
Effective context is an operational estimate, not a claim that model architecture changes by language. Cost uses GPT-4o input pricing of $2.50 per million tokens as of May 2026; the token ratios are the durable result.
View full context table (Table 2)
| Language | Premium / 1k calls | Effective context |
|---|---|---|
| English | $0.00 | 128,000 (100%) |
| Arabic | $0.06 | 89,148 (70%) |
| Hindi | $0.09 | 74,461 (58%) |
| Bengali | $0.07 | 81,967 (64%) |
| Tamil | $0.14 | 61,121 (48%) |
| Yoruba | $0.18 | 53,951 (42%) |
The premium propagates when multilingual data is built.
A 100,000-token English instructional corpus expands non-uniformly when represented as equivalent multilingual content using GPT-4o TFRs.
Longer tokenized representations increase storage, transmission, batching, and curation burden before downstream model quality is even considered — a structural barrier to equitable resource development that operates independently of model quality.
Token efficiency is not answer quality.
TEA is deliberately an infrastructure audit. It does not claim that a lower TFR guarantees a better model response.
120 items in one technical-education domain — a workshop-scale audit, not a universal tokenizer ranking.
Bengali and Hindi are the human-validated primary cases; Arabic, Tamil, and Yoruba are exploratory.
The code-switching analysis is currently Bengali-only.
Dollar figures are illustrative and provider-dependent. TFR is the stable quantity.
Make the invisible denominator visible.
Four reporting practices the paper proposes — none of which require training a new model.
Model cards and multilingual benchmark papers should include language-level tokenization statistics, especially for underserved languages.
Context-window claims should be paired with language-adjusted estimates whenever multilingual deployment is intended.
Programming education, legal assistance, health guidance, and other terminology-heavy domains should be audited independently, since code-switching and domain vocabulary change tokenization behavior.
Non-uniform token scaling (100k → 156k Bengali → 209k Tamil → 237k Yoruba) increases storage, transmission, and curation cost for underserved languages — a structural barrier that operates independently of model quality.
Paper, benchmark, and selected literature cited by TEA.
These references are drawn from the paper's bibliography and linked to primary publication or preprint pages where available.
Reproduce the benchmark
Corpus, token counts, scripts, figures, result tables, and run instructions.
github.com/HeyAvijitRoy/tea-benchmarkCite this work
If you use this work, please cite the paper. Cite the poster separately if you are referencing the LM4UC 2026 presentation itself.
Paper Citation
@article{roy2026tokenizationpremium,
title = {Measuring the Tokenization Premium: A Cost Audit for Underserved Language Communities},
author = {Roy, Avijit and Roy, Proma and Patel, Hrishitva},
year = {2026},
journal = {arXiv preprint arXiv:2608.09046},
doi = {10.48550/arXiv.2608.09046},
url = {https://doi.org/10.48550/arXiv.2608.09046},
note = {Accepted for oral presentation and poster, Language Models for Underserved Communities (LM4UC) Workshop, IJCAI 2026}
}
Poster Citation
@misc{roy2026tokenizationpremium_poster,
author = {Roy, Avijit and Roy, Proma and Patel, Hrishitva},
title = {Measuring the Tokenization Premium: A Cost Audit for Underserved Language Communities (Poster)},
year = {2026},
month = aug,
doi = {10.13140/RG.2.2.25750.31049},
url = {https://doi.org/10.13140/RG.2.2.25750.31049},
note = {Poster presented at the Language Models for Underserved Communities (LM4UC) Workshop, IJCAI 2026}
}