UnicodeDecodeError (Python) and mojibake

In short: Python cannot read bytes as text because the file has a different character encoding than assumed.

In more detail: Message: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 12: invalid continuation byte or on Windows 'charmap' codec can't decode byte …. Text consists of bytes; an encoding defines which bytes correspond to which characters. If the assumed one does not match the actual one, this error or mojibake (“ä” instead of “ä”) arises.

In Depth

Causes

• The file is Latin-1/Windows-1252 (for example from older Excel or Windows), Python expects UTF-8.
• On Windows the default encoding of open() is often cp1252, not UTF-8.
• Binary data (images, ZIP) is read as text.
• A UTF-8 file with BOM (byte order mark) or UTF-16.
• Text wrongly re-encoded several times.

Solutions

1. Specify the encoding: open("file.txt", encoding="utf-8") or encoding="cp1252" / "latin-1".
2. Detect the encoding: editor (VS Code: bottom right), file -i file.txt, library charset-normalizer.
3. With BOM: encoding="utf-8-sig".
4. For binary data open in "rb" mode.
5. As a last resort replace errors: errors="replace" or "ignore" (loses characters, only if acceptable).
6. Standardise everything on UTF-8, save files as UTF-8 and set <meta charset="utf-8"> in HTML. For CSV from Excel: choose “CSV UTF-8” or read cp1252.

Repairing mojibake

Typical pattern: UTF-8 text was read as Latin-1. Way back in Python: text.encode("latin-1").decode("utf-8"). The library ftfy helps. Always test with examples, because multiple mis-encodings are more complicated.

See also: ASCII and character sets, UTF-8 and character encoding, ModuleNotFoundError