Tokenisation: BPE and vocabulary

In short: A language model does not read letters or whole words but tokens: frequent word pieces. The tokenizer splits text into tokens and assigns each a number.

In more detail: The vocabulary usually has 32,000 to 250,000 entries. Frequent words are one token, rare ones are split into pieces, unknown ones in an emergency into single bytes. So there is never an unknown word.

In detail

• BPE (byte-pair encoding): start with single characters/bytes. Repeatedly merge the most frequent adjacent pair into a new token until the vocabulary is full. This procedure is learned from the training text.
• Example: “Unbelievable” → Un believ able. An English word is often 1 token, a long German word 3 to 5.
• Rule of thumb: English about 4 characters or ¾ word per token, German fewer characters per token, code and special characters likewise.
• Tools: SentencePiece, tiktoken, Hugging Face Tokenizers.
• Consequences: arithmetic, counting letters (“how many r in strawberry”) and rhymes are hard because the model sees tokens, not letters.
• Costs: APIs charge per token, see Token (AI).
• Special tokens: start, end, role markers (<|user|>).
• Multilinguality: languages with less training text often need more tokens per sentence and are therefore more expensive.

See also: Token (AI), Embedding (vector representation)