Token (AI)

In short: A token is the small piece of text into which a language model splits text. A token is often a word fragment, a short word or a punctuation mark.

In more detail: In German a token is roughly 3 to 4 letters, in English about 4. 1,000 tokens are about 700 to 750 English words; German texts need somewhat more tokens because of long compound words.

Our context: In the AI speed calculator speed is given in tokens per second and converted into words and book pages per minute.

In Depth

• Tokenizer: a method (for example byte-pair encoding) that merges the most common character sequences into tokens.
• Why tokens: The model works with numbers. Each token has a number and a vector (Embedding).
• Cost: Providers bill input and output tokens.
• Context window: the maximum number of tokens a model considers at once (Context window).

See also: Tokens per second, Context window