Tokeniser explorer

Models do not see words. Type in two languages and compare how many tokens each needs; then lower the number of merges and watch the pieces get smaller. Everything runs in this page after the one-time load of the bundled vocabulary: nothing you type leaves your browser.

Text Text to tokenise (one sentence per line)

Vocabulary Merges applied

The bundled vocabulary learned 4 000 merge rules from English and Norwegian text. At 4 000, common chunks such as whole short words become single tokens. Drag the slider down and the model is only allowed to use the earliest, most frequent merges, so words break back down into smaller pieces, and at 0 every token is a single raw byte.

With "Show bytes" on, click or tab to any token below to see the raw byte values it is made of.

Click or focus a token to see its bytes.

What to notice: the Norwegian sentence almost always needs more tokens than the English one, because the vocabulary saw far less Norwegian text during training. Letters such as å, æ and ø are two bytes each in UTF-8, so at low merge counts they show up as more than one token.