Unicode and Python 3

Ezio Melotti

Something about me

• 24 years old
• Information Technology (IT) student
• I'm studying at the Turku University of Applied Science...

Something about me

Outline

• Introduction: Character sets and encodings
• Unicode in Python 2 and Python 3
• Unicode and the real world
• Q&A

ASCII

• Limited to 128 chars (7 bits)
  • not 256!

 !"#$%&\'()*+,-./0123456789:;<=>?@
ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`
abcdefghijklmnopqrstuvwxyz{|}~

• 8^th bit first used for parity checking
  • Then used to represent more chars

8-bits Character Sets

• 8-bit charsets → 256 chars:
  • all the 128 ASCII chars
  • + 128 more chars, for example:

 !"#$%&\'()*+,-./0123456789:;<=>?@
ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`
abcdefghijklmnopqrstuvwxyz{|}~

 ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿
ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞß
àáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ

• Lot of different character sets
  • different chars in the range 128-255...

ISO-8859-* Family

Unicode

• Unicode provides a unique number for every character
  • no matter what the platform
  • no matter what the program
  • no matter what the language
• 1114112 different characters
  • U+0000 to U+FFFF → BMP (Basic Multilingual Plane)
  • U+10000 to U+10FFFF
• Replaces hundreds of existing character sets

Unicode characters

 !"#$%&\'()*+,-./0123456789:;<=>?@
ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`
abcdefghijklmnopqrstuvwxyz{|}~

 ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿
ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞß
àáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ

ĀāĂ㥹ĆćĈĉĊċČčĎďĐđĒēĔĕĖėĘęĚěĜĝĞğ
ĠġĢģĤĥĦħĨĩĪīĬĭĮįİıIJijĴĵĶķĸĹĺĻļĽľĿ
ŀŁłŃńŅņŇňʼnŊŋŌōŎŏŐőŒœŔŕŖŗŘřŚśŜŝŞş
ŠšŢţŤťŦŧŨũŪūŬŭŮůŰűŲųŴŵŶŷŸŹźŻżŽž

How many characters?

512 chars per slide → 2175 slides to represent them all

Unicode

Character Sets and Encoding

• Character Sets:
  • Mappings from a set of abstract chars to a set of integers
  • Example: 'a' ↔ 97

• Character Encoding:
  • Mappings from a Character set to a sequence of bits
  • Example: 'a' ↔ 01100001 (\x61)

Codepoints

• An integer in the range from 0 to 10FFFF
• For example 'a' → U+0061 (97 in base 10)
• Characters (e.g. ☃) have:
  • a codepoint (e.g. U+2603)
  • a name (e.g. SNOWMAN)
  • a category (e.g. So - Symbol, Other)
  • a block (e.g. Miscellaneous Symbols)
  • and other attributes

Unicode Transformation Format (UTF)

• UTF family: UTF-8, UTF-16, UTF-32
  • can represent all the Unicode chars
  • multibyte character encodings
    • UTF-8 → 1, 2, 3 or 4 bytes
    • UTF-16 → 2 or 4 bytes
    • UTF-32 → 4 bytes
• UTF-8
  • always use UTF-8 for your data
  • default encoding in lot of places
  • if you don't know the encoding, assume UTF-8

Unicode strings

>>> unistr = u'I am a Unicode string on Python 2'
>>> type(unistr)
<type 'unicode'>

>>> unistr = 'I am a Unicode string on Python 3'
>>> type(unistr)
<class 'str'>

Escape sequences in string literals

>>> u'☃' == u'\u2603' == u'\U00002603' == u'\N{SNOWMAN}'
True

>>> '☃' == '\u2603' == '\U00002603' == '\N{SNOWMAN}'
True

Unicode is "abstract"

• Unicode strings contain “abstract” text
  • Indipendent from the byte representation
  • All the characters can be used
• But it is not possible to:
  • Save them on a disk
  • Send them on the network
• We also need...

Byte strings

>>> bytestr = 'I am a Byte string on Python 2'
>>> type(bytestr)
<type 'str'>

>>> bytestr = b'I am a Byte string on Python 3'
>>> type(bytestr)
<class 'bytes'>

String types in Python 2 and 3

Unicode in Python 3

• Unicode strings are now the default
• The u is no longer necessary for Unicode strings
• The b is now necessary for Byte strings

Encoding/Decoding

• Encoding:
  • Unicode → Bytes

  • >>> bytestr = unistr.encode('utf-8')

• Decoding:
  • Bytes → Unicode

  • >>> unistr = bytestr.decode('utf-8')

Encoding/Decoding - Errors

>>> snowman = 'Snowman: ☃' # Python 3
>>> snowman.encode('ascii', 'strict') # default
UnicodeEncodeError: 'ascii' codec can't encode character
  '\u2603' in position 9: ordinal not in range(128)
>>> snowman.encode('ascii', 'replace')
b'Snowman: ?'
>>> snowman.encode('ascii', 'ignore')
b'Snowman: '
>>> snowman.encode('ascii', 'xmlcharrefreplace')
b'Snowman: ☃'
>>> snowman.encode('ascii', 'backslashreplace')
b'Snowman: \\u2603'

>>> snowman.encode('utf-8')
b'Snowman: \xe2\x98\x83'

Implicit Decoding on Python 2

• Python 2 allows to mix Unicode and Byte strings:

  >>> u'Unicode and ' + 'Bytes'
  u'Unicode and Bytes'

  • sys.getdefaultencoding() is used
  • On Py2 sys.getdefaultencoding() == 'ascii'
  • if the 'str' contains non-ascii the decoding fails

    >>> u'Unicode and ' + 'ByteSnowMan: ☃'
    UnicodeDecodeError: 'ascii' codec can't
     decode byte 0xe2 in position 13: ordinal
     not in range(128)

• On Python 3 they can NOT be mixed:

  >>> 'Unicode and ' + b'Bytes'
  TypeError: Can't convert 'bytes' object to str implicitly

Encoding/Decoding - UnicodeErrors

• UnicodeError
  • UnicodeEncodeError: raised during encoding
  • UnicodeDecodeError: raised during decoding

>>> unistr = 'Minä tykkään Unicodesta!' # Python 3
>>> unistr.encode('ascii')
UnicodeEncodeError: 'ascii' codec can't encode character
  '\xe4' in position 3: ordinal not in range(128)
>>> bytestr = unistr.encode('iso-8859-1')
>>> bytestr.decode('utf-8')
UnicodeDecodeError: 'utf8' codec can't decode bytes
  in position 3-5: invalid data

When to use Unicode and Byte strings

1. Decode the input as soon as you receive it
2. Work only with Unicode
3. Encode the output just before sending it

Know the Encoding

• You have to know the correct encoding
  • Otherwise you can't decode
• You could guess...
  • chardet module
    • auto-detects the encoding
  • But it's not 100% reliable

When to use Unicode and Byte strings

• Unicode
  • When working with the text
  • splitting, slicing, joining, adding, removing, uppercasing, lowercasing...
• Byte strings
  • ONLY during the Input/Output
  • reading/writing from files, sockets, ...

Why?

>>> unistr = u'Minä tykkään Unicodesta!' # Python 2
>>> # I ♥ Unicode
>>> bytestr = unistr.encode('utf-8')
>>> print len(unistr)
24
>>> print len(bytestr)
27
>>> print unistr[:4]
Minä
>>> print bytestr[:4]
Min├
>>> bytestr
'Min\xc3\xa4 tykk\xc3\xa4\xc3\xa4n Unicodesta!'

Where?

Source Encoding

• PEP-0263: Defining Python Source Code Encodings

  # -*- coding: utf-8 -*-
  print u'This file is saved in UTF-8 ☺'

• Related only to the text in the source file
  • not to the encoding that your program will handle
  • allows to insert non-ASCII chars in string literals and comments
    • in Python 3 in variables names too (PEP-3131)
• If not specified, the default is:
  • ASCII on Python 2 (PEP-0263)
  • UTF-8 on Python 3 (PEP-3120)

Web Pages

• Py2: urllib.urlopen → Bytes
• Py3: urllib.request.urlopen → Bytes

• xml.dom.minidom ↔ Bytes/Unicode

• Beautiful Soup → Unicode

Web Pages - Find the Encoding

• Specified in HTTP headers:

  content-type: text/html; charset=utf-8

• XML declarations:

  <?xml version="1.0" encoding="utf-8" ?>

• HTML <meta> tags:

  <meta http-equiv="Content-Type"
   content="text/html; charset=utf-8">

Files

• Python 2:
  • open('file.txt', 'rb') → Bytes
  • open('file.txt', 'rt') → Bytes
  • codecs.open('file.txt', 'rt', 'utf-8') → Unicode
• Python 3:
  • open('file.txt', 'rb') → Bytes
  • open('file.txt', 'rt') → Unicode
  • codecs.open() no longer needed

Files - 2

• Py3's open() accepts the encoding and errors args: open('file.txt', 'rt', 'utf-8', 'strict')
• If encoding=None, Python:
  • tries to determine the encoding
  • uses sys.getdefaultencoding()

os module

• Py2: os.getcwd()/os.getcwdu() → Bytes/Unicode
• Py3: os.getcwdb()/os.getcwd() → Bytes/Unicode

• Py2: os.listdir('')/os.listdir(u'') → Bytes/Unicode
• Py3: os.listdir(b'')/os.listdir('') → Bytes/Unicode

os module

>>> os.getcwd()
'/home/wolf'
>>> os.getcwdu()
u'/home/wolf'
>>> os.listdir('/home')
['wolf', '.directory']
>>> os.listdir(u'/home')
[u'wolf', u'.directory']

>>> os.getcwdb()
b'/home/wolf'
>>> os.getcwd()
'/home/wolf'
>>> os.listdir(b'/home')
[b'wolf', b'.directory']
>>> os.listdir('/home')
['wolf', '.directory']

Network

• socket → Bytes
• ftplib → Bytes
• Py2: httplib
  • Headers → Bytes
  • Content → Bytes
• Py3: http.client
  • Headers → Unicode
  • Content → Bytes

Terminal

 >>> os.environ['LANG']
'en_US.UTF-8'
>>> sys.stdout.encoding
'UTF-8'
>>> sys.stdin.encoding
'UTF-8'
>>> sys.stdout.errors # Python 3 only!
'strict'
>>> sys.stdout.encoding = 'ascii'
AttributeError: can't set attribute
>>> sys.stdout.errors = 'replace'
AttributeError: can't set attribute

Terminal - 2

• sys.stdin/sys.stdout can be replaced with other file-like objects
• Not all the terminals can handle UTF-8
  • for example, Windows terminal can't

String representation in Python 3

>>> unistr = u'Minä tykkään Unicodesta!'
>>> print repr(unistr)
u'Min\xe4 tykk\xe4\xe4n Unicodesta!'

>>> unistr = 'Minä tykkään Unicodesta!'
>>> print(repr(unistr))
'Minä tykkään Unicodesta!'

Python 2 → Python 3

• On Python 2:
  • use unicode for all unencoded text
  • use str only for binary or encoded data
  • avoid implicit decoding
    • always encode/decode explicitely
• The 2to3 tool
  • removes the u from unicode strings
  • does not add the b to byte strings
  • converts unicode(bytestr, enc) to str(bytestr, enc)

Other things

• unicodedata module
• bytearray mutable type (PEP-3137)
• PEP-3131: Supporting Non-ASCII Identifiers
• UTF-16 vs. UTF-32 Python builds

The end