Tokenizer pitfalls: numbers, code and emoji

In short: Language models do not see text but tokens. Because the tokenizer splits words, numbers and characters differently, typical traps appear: arithmetic mistakes, wrongly counted letters and uneven costs per language.

Why it matters

A tokenizer (see Tokenisation, BPE and vocabulary) cuts text into frequent pieces. The model learns the pieces, not the single letters inside them.

• Numbers: “12345” may be split into “123” and “45” or into single digits. Depending on the split the model calculates better or worse. Modern tokenizers therefore often split numbers into fixed digit blocks.
• Counting letters: “How many r in strawberry?” is hard because “strawberry” may be only two tokens and the letters are not directly visible to the model.
• Code: indentation, spaces and rare identifiers cost many tokens. A tab or four spaces is one or several tokens depending on the tokenizer.
• Emoji and rare scripts: an emoji is several bytes in UTF-8 and can cost several tokens. For rare languages text is often much longer in tokens than in English, so more expensive and shorter in the context window.
• Spaces: “Hello” and ” Hello” (with a leading space) are often different tokens.

What you can do

1. Count tokens before estimating cost (the vendors’ tokenizer libraries).
2. For calculations and counting let the model use a tool (calculator, code), see Tool use, agents and MCP.
3. Separate important numbers and names clearly in the prompt (lines, delimiters).

See also: Context window, Tokenisation, BPE and vocabulary