UTF-16 and surrogate pairs

In short: UTF-16 stores Unicode characters in 16-bit units. Characters beyond the first 65,536 need two units, a surrogate pair.

In more detail: Java, JavaScript, C# and Windows use UTF-16 internally. That is why a string’s length there often is not the number of visible characters.

In Depth

• Basic plane (BMP): Code points U+0000 to U+FFFF take one unit.
• Surrogate pair: A code point from U+10000 (many emoji) is split into a high (D800 to DBFF) and a low unit (DC00 to DFFF).
• Consequence: "😀".length is 2 in JavaScript. Cutting in the middle of a pair yields an invalid character. Array.from(s) or for…of work per code point.
• Byte order: UTF-16LE and UTF-16BE, often with a BOM (byte order mark) at the file start.
• Versus UTF-8: UTF-8 is ASCII-compatible and the web standard. UTF-16 saves little for most CJK text.
• Finer still: One visible character can consist of several code points (grapheme clusters, such as flags or skin-tone emoji).

See also: UTF-8 and character encoding