UTF-8 explained: how text is stored, and why it turns into é and ’
Every text file is bytes, and every program that opens one has to decide which characters the bytes stand for. For decades they guessed differently, which is why an email says "don’t" and a CSV turns "café" into "café". UTF-8 ended the guessing for anyone who uses it consistently. This guide explains what it is and how the garbage happens; the HTML entities tool and the hex viewer let you see the bytes.
Unicode: numbers for every character
Unicode assigns a number — a code point — to every character in every writing system, plus symbols and emoji: A is U+0041, é is U+00E9, 漢 is U+6F22, 😀 is U+1F600. There are over 150,000 assigned and room for a million. Unicode says nothing about bytes; it is only the catalogue. Before it, each language area had its own 256-character table (Latin-1, Windows-1252, Shift-JIS, KOI8-R), and a file carried no marker of which one it used.
UTF-8: turning numbers into bytes
UTF-8 (1992, Ken Thompson and Rob Pike) encodes code points as one to four bytes. The first 128 — plain ASCII — are one byte each, identical to the old ASCII, so every ASCII file is already valid UTF-8 and English text costs nothing extra. Latin accents, Greek and Cyrillic take two bytes; most Asian scripts three; emoji four. The byte patterns are self-synchronising — you can tell from any byte whether it starts a character — which makes UTF-8 robust and searchable, and it is now the encoding of over 98% of web pages and the default of every modern operating system and language. Its cousins UTF-16 (two or four bytes, used inside Windows and JavaScript) and UTF-32 encode the same code points differently.
Why text turns into é and ’
Mojibake is what you see when bytes written in one encoding are read in another. "é" in UTF-8 is the two bytes C3 A9; read them as Windows-1252 and you get "é" — two characters. The curly apostrophe (U+2019) is E2 80 99, which reads as "’". The reverse mistake — Windows-1252 bytes read as UTF-8 — produces question marks or replacement characters (�) because the bytes aren't valid UTF-8 sequences. Once you recognise the signatures you can diagnose most cases by eye: Ã-something means UTF-8 read as Latin; � means the opposite.
Fixing encoding problems
- CSV from Excel: Excel writes and reads CSVs in the system's legacy encoding unless told otherwise. Save as "CSV UTF-8" in Excel, or start the file with a byte-order mark (BOM, the bytes EF BB BF), which Excel takes as a signal — Excel to CSV and CSV vs Excel: what each file really contains, and what breaks in between handle this.
- Files in an editor: reopen with the correct encoding (most editors offer "reopen with encoding"), then save as UTF-8.
- Already-corrupted text: if the damage is a single mis-decode, re-encoding the mojibake as Windows-1252 and decoding as UTF-8 recovers it; double corruption doesn't.
- Databases and APIs: declare UTF-8 end to end — column collation, connection charset, HTTP header. The problem is almost always one hop that defaulted to something else.
Encoding on the web
HTML5 assumes UTF-8; declare it anyway with <meta charset="utf-8"> in the first 1024 bytes and a Content-Type header, because the fallback on a mismatch is guessing. HTML entities (é for é, ’ for ’) exist so characters can be written in pure ASCII when needed — in a template with an unknown encoding, or for the five characters (< > & " ') that mean something in HTML. Base64 is a different thing: it turns arbitrary bytes into ASCII for transport, not characters into bytes (What Base64 is, and what it is not). URL encoding does the same job for addresses, where UTF-8 bytes become %C3%A9.
Sources and further reading
The claims in this guide rest on these references, which were checked when the guide was last updated. Spotted an error? The contact page says how to report it.