Tokenizer Lens
Language models don't read letters or words — they read tokens. Type English or Japanese and watch five real tokenizers carve it up, right in your browser.
Each chip is one token. A chip with a small number is a single character the tokenizer had to spell out in raw bytes — one character, several tokens. ·marks a space.
What you’re looking at
Before a language model sees your sentence, a tokenizer chops it into pieces from a fixed vocabulary — anywhere from 30,000 to 250,000 entries. Each piece becomes a number, and the model only ever works with those numbers. Everything a model “knows” about spelling, rhyme, or kanji has to be learned through that filter.
Each row above is a real tokenizer, downloaded from the Hugging Face Hub and run entirely in your browser — nothing you type leaves your device.
Things to try
- English vs. Japanese. Paste the same idea in both languages and compare the counts. Older vocabularies were built mostly from English text, so Japanese gets split into far more, far smaller pieces. That means more tokens per sentence, which means higher API cost and less room in the context window for the same meaning.
- Byte fragments. Watch for chips marked with a small number. Those are single characters (often kanji) that the tokenizer had no entry for, so it fell back to raw UTF-8 bytes — one character, two or three tokens.
- Numbers and spacing. Try
1234567or a string of spaces. Different tokenizers group digits very differently, which is one reason models used to be bad at arithmetic.
What I learned building it
- The newer tokenizers (GPT-4o, Gemma, Llama 3) handle Japanese dramatically better than the 2023-era ones — common kana endings like っています often land in a single token.
- BERT, a 2018 model, simply has no idea what many kanji are: it marks them
[UNK], unknown. That’s a vivid picture of how far multilingual AI has come in a few years. - The “Claude” row is Anthropic’s 2023 tokenizer, the one it published openly. Current Claude models use newer tokenizers that aren’t public, so treat that row as history, not a live measurement.