Understanding Encodings

Ezio Melotti

Something about me

• Python Core Developer since June 2009
• Python Programming lecturer at the Turku University of Applied Sciences
• Presented several talks about Unicode and The development process of Python at EuroPython, PyCon FI, PyCon IT
• Member of the Italian Mars Society

As Joel Spolsky said

"If you are a programmer working in 2003 and you don't know the basics of characters, character sets, encodings, and Unicode, and I catch you, I'm going to punish you by making you peel onions for 6 months in a submarine.

I swear I will."

Joel Spolsky, The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)

In a submarine...

by mnorri

...peeling onions

by वंपायर

Outline

• Character sets
• Encodings
• Recommendations and common problems
• Q&A

What is a Character Set?

A Character Set is a collection of elements used to represent textual information.

What is a Character Set?

• Most of the Character Sets assign a number to each element - they are also known as Coded Character Sets

Still not enough...

What if I want to mix русский and العربية?

What if I want to use ᐃᓄᒃᑎᑐᑦ (inuktitut)?!

en.wikipedia.org/wiki/File:IqaluitStop.jpg

We need a better solution...

ISO-8859-* Family

ASCII, ISO-8859-1, ISO-8859-5, ISO-8859-5

Introducing Unicode

Unicode

Introducing Unicode

Unicode

Introducing Unicode

• Unicode covers all the characters for all the writing systems of the world, modern and ancient.
• Unicode provides a unique number for every character
  • no matter what the platform
  • no matter what the program
  • no matter what the language
• The number is called "codepoint".
• 1114112 different codepoints
  • U+0000 to U+10FFFF
• Replaces hundreds of existing character sets

Codepoints

• An integer in the range from 0 to 10FFFF
• Expressed with the notation U+XXXX
• For example 'a' → U+0061, 'ä' → U+00E4
• Each Unicode character (e.g. ☃) has:
  • a codepoint (e.g. U+2603)
  • a name (e.g. SNOWMAN)
  • a category (e.g. So - Symbol, Other)
  • a block (e.g. Miscellaneous Symbols)
  • and other attributes

Unicode Planes

Unicode is organized in 16 planes, with 65536 codepoints each

• Basic Multilingual Plane (BMP)
  • Plane 0: U+0000–U+FFFF
  • includes most of the commonly used characters
• Supplementary Planes (non-BMP)
  • Plane 1 (U+10000–U+1FFFF): Supplementary Multilingual Plane
  • Plane 2 (U+20000–U+2FFFF): Supplementary Ideographic Plane
  • Planes 3–13 (U+30000–U+DFFFF): Unassigned
  • Plane 14 (U+E0000–U+EFFFF): Supplementary Special-purpose Plane
  • Planes 15–16 (U+F0000–10FFFF): Private Use Area

Unicode characters

 !"#$%&\'()*+,-./0123456789:;<=>?@
ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`
abcdefghijklmnopqrstuvwxyz{|}~

 ¡¢£¤¥¦§¨©ª«¬ ¯°±²³´µ¶·¸¹º»¼½¾¿
ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞß
àáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ

ĀāĂ㥹ĆćĈĉĊċČčĎďĐđĒēĔĕĖėĘęĚěĜĝĞğ
ĠġĢģĤĥĦħĨĩĪīĬĭĮįİıIJijĴĵĶķĸĹĺĻļĽľĿ
ŀŁłŃńŅņŇňʼnŊŋŌōŎŏŐőŒœŔŕŖŗŘřŚśŜŝŞş
ŠšŢţŤťŦŧŨũŪūŬŭŮůŰűŲųŴŵŶŷŸŹźŻżŽž

What is a Character Set?

A Character Set is a collection of elements used to represent textual information.

Encoding types

Encodings can be divided in:

Single-byte encodings

• Use only 1 byte, limited to 256 characters

Multi-byte encodings

• Use more than 1 byte, and are divided in
  • Fixed width: uses a fixed number of bytes for each character
  • Variable width: uses a variable number of bytes

8-bits/single-byte Encodings

For 8-bits character sets, the terms "character set" and "encoding" might overlap

• 256 possible chars, 256 values representable with 1 byte (8 bits)
• Single-byte encodings
• For example, ISO-8859-1 is both a character set and an encoding

Cause of endless confusions:

<meta charset="utf-8">

this should actually be called 'encoding'

Unicode Transformation Format (UTF)

UTF family: UTF-8, UTF-16, UTF-32

• can represent all the Unicode codepoint
• multibyte character encodings
  • UTF-8 → 1, 2, 3 or 4 bytes
  • UTF-16 → 2 or 4 bytes
  • UTF-32 → 4 bytes

UTF-8

• always use UTF-8 for your data
• default encoding in lot of places
• if you don't know the encoding, assume UTF-8

UTF-32

UTF-32 is a fixed-width encoding

• each codepoint is encoded with 4 bytes
• not very memory-efficient
• easier to work with (indexing, slicing)

Recommendations

Remember:

• Decode early
• Work with Unicode only
• Encode late

and:

• use always UTF-8 for your data
• never mix text and bytes
• never mix encodings
• never do text processing on bytes
• always know the encoding

Encoding/Decoding - UnicodeErrors

UnicodeError:

UnicodeEncodeError: raised during encoding
UnicodeDecodeError: raised during decoding

>>> unistr = 'Minä tykkään Unicodesta!' # Python 3
>>> unistr.encode('ascii')
UnicodeEncodeError: 'ascii' codec can't encode character
  '\xe4' in position 3: ordinal not in range(128)
>>> bytestr = unistr.encode('iso-8859-1')
>>> bytestr.decode('utf-8')
UnicodeDecodeError: 'utf8' codec can't decode bytes
  in position 3-5: invalid data

Source Encoding

PEP-0263: Defining Python Source Code Encodings

# -*- coding: utf-8 -*-
print u'This file is saved in UTF-8 ☺'

Related only to the text in the source file

• not to the encoding that your program will handle
• allows to insert non-ASCII chars in the source code

If not specified, the default is:

• UTF-8 on Python 3 (PEP-3120)
• ASCII on Python 2 (PEP-0263)

Mojibake

Mojibake (文字化け): "unintelligible sequence of characters"

• 8-bit encoding can encode/decode everything
• UTF-8 uses '�' in case of errors

Narrow vs Wide

On Python <3.3 there are two different Python builds:

Fixed in Python 3.3! Thanks to PEP 393

PEP 393: Flexible String Representation

• Number of bytes per codepoint determined by the highest codepoint:
  • U+0000–U+00FF: 1 byte
  • U+0100–U+FFFF: 2 bytes
  • U+10000–U+10FFFF: 4 bytes
• Uses less memory
• Still fast (sometimes even faster!)
• Same behavior of wide builds at Python-level
• Backward compatible on the C-level (but you should switch to the new API)
• http://www.python.org/dev/peps/pep-0393/

The end

Questions? ☃