ASCII and character sets: How text arises in the computer

In short: Computers store only numbers; a character set defines which number corresponds to which letter. ASCII was the beginning, Unicode is today’s standard.

In more detail: When you type an “A”, the computer stores a number. For all programs to read the same number as “A”, a mapping table is needed: the character set. Different character sets explain why umlauts sometimes appear as gibberish (“ä”).

In Depth

ASCII

ASCII (American Standard Code for Information Interchange, 1963) uses 7 bits, i.e. 128 characters: capital letters A–Z (65–90), lower-case a–z (97–122), digits 0–9 (48–57), punctuation and control characters such as line feed (10) and tab (9). Umlauts and special characters are missing. To remember: A = 65, a = 97, difference 32 (one bit).

Extensions and chaos

The eighth bit allowed 128 more characters. Every region filled them differently: ISO-8859-1 (Latin-1) for Western Europe with umlauts, Windows-1252, ISO-8859-5 (Cyrillic) etc. The same number meant a different character depending on the table, and texts from other languages could not be mixed.

Unicode and UTF-8

Unicode gives every character in the world (over 150,000, including emoji) a unique number, the code point (for example U+00E4 for “ä”). UTF-8 stores these code points with 1 to 4 bytes: ASCII characters remain 1 byte (compatible with ASCII), umlauts need 2, many special characters 3, emoji 4. UTF-8 is today’s standard on the web (over 98 % of all pages) and in most programming languages.

Practice and error patterns

• Always save files and pages as UTF-8, set <meta charset="utf-8"> in HTML.
• “ä” instead of “ä” arises when UTF-8 text is read as Latin-1 (mojibake).
• The length of a text in characters and in bytes are different things.
• With ord('A') in Python you get the code point (65), with chr(65) the character back.
• Line breaks: Unix \n, Windows \r\n.

See also: UTF-8 and character encoding, Unicode, Binary, decimal, hexadecimal, char (character)