From-scratch Byte-Pair-Encoding · India Wikipedia article

Multilingual BPE Tokenizer

One shared vocabulary of tokens across English, Hindi, Telugu & Kannada.

Assignment self-score

Score rewards an even tokenizer: it is highest when all four languages have the same tokens-per-word ratio (X).

Per-language ratios — X = tokens ÷ words

Recomputed by encoding each article's words with the shipped tokenizer.json. Target per assignment: X ≤ 1.2.

Language	Article words	Tokens	X (tok/word)	≤1.2?

How the score is computed

Vocabulary composition

Base alphabet = single characters; the remaining tokens are learned merges. Breakdown by script:

Explore & download the tokenizer

All vocabulary tokens are embedded in this page. </w> marks a word-final token (a whole word that became one token). Search filters live.

Reproduce these numbers yourself

Load tokenizer.json, encode each language's words (a word = \p{L}[\p{L}\p{M}]*) with the learned merges, then X_lang = total tokens ÷ total words. The four X values above are produced exactly this way; the build step asserts they equal the shipped summary.json before publishing. Score = 1000 ÷ (X_max − X_min).