Building a Multilingual BPE Tokenizer for India's Wikipedia Page
A tokenizer chops text into the pieces a language model actually reads. The challenge here: build one from scratch — no libraries — that shares a single 10,000-token vocabulary across four very different scripts, using the "India" Wikipedia article in English, Hindi, Telugu, and Kannada.
The metric: fertility
Quality per language is measured as fertility — the average number of tokens a word gets split into:
X = total tokens ÷ total words
If a word becomes a single token, X = 1.0 (perfect). If it shatters into pieces, X climbs. The assignment's target is X ≤ 1.2, and the final score rewards evenness across languages:
score = 1000 ÷ (X_max − X_min)
That formula is the whole game. A tokenizer that's brilliant at English but clumsy at Telugu scores badly, because the spread between its best and worst language is large. The goal is not just low fertility — it's the same fertility everywhere.
How BPE works
Byte-Pair Encoding starts from individual characters and repeatedly glues together the most frequent adjacent pair, one merge at a time. Each merge you keep is one slot in the vocabulary. More merges → longer pieces → fewer tokens per word. The 10,000 budget is simply how many merges you're allowed — and all four languages compete for the same 10,000 slots.
The result — explore it live
The widget below shows every ratio, the token statistics, the score calculation, and a searchable list of all 10,000 tokens (with downloads). Every number in it is recomputed by loading the trained tokenizer and re-encoding the actual article text — the exact procedure used to verify a submission — so what you see is what a grader reproduces.
The interesting part: why the score caps out
You'd expect that with enough tuning, all four languages could be pushed to X ≈ 1.0 and the spread driven to zero. They can't — and the reason is instructive:
- Small corpora are all-or-nothing. The Telugu (2,410 words) and Kannada (905 words) articles are tiny and homogeneous. Either the vocabulary memorizes all their words at once (X = 1.0) or it barely touches them (X ≈ 1.4–2.0). There is no smooth middle — their words are all equally rare, so they cross the "worth a merge slot?" threshold together.
- The budget is conserved. Spend merges memorizing Telugu, and those slots come straight out of English and Hindi — whose fertility then rises. The peak just moves from one language to another; the gap stays put.
So the maximum fertility sits near ~1.43 in every configuration, and the spread lands around 0.435 no matter how you weight the languages. Beating it meaningfully would need a different mechanism — reserving a fixed merge budget per language rather than letting them fight over a shared pool.
Reproduce it yourself
Load tokenizer.json (downloadable from the widget), encode each language's words with the learned merges, and compute tokens ÷ words per language. The four ratios above are produced exactly that way — no hand-entered numbers.