The Development Process of Python

Helsinki Python Sprint

2014-08-02 / 2014-08-03

Something about me

  • Python Core Developer since June 2009
  • Information Technology student at the Turku University of Applied Sciences
  • Sometimes also Python Programming lecturer
  • Maintainer of the "html" package and of the Python bug tracker
  • Co-maintainer of a number of other things
  • Member of the Italian Mars Society

Outline

  • Community
  • Tools
  • HG
  • Workflow
  • CPython

Community

  • Developers
  • Mailing lists
  • IRC channels
  • Conferences

Developers

Mailing Lists (MLs)

  • python-dev the "heart of Python's development"
  • core-mentorship help for contributors
  • python-ideas where new ideas are proposed
  • python-committers discussions for committers only
  • python-checkins where all the commits are reported
  • python-bugs-list and new-bugs-announce where the changes in the issue tracker are reported

IRC channels

#python-dev

  • the official IRC channel about Python development
  • some of the core developers hang around there
  • irker reports updates in the tracker and commits
  • py-bb reports buildbots status

#python-infra

  • for infrastructure-related issues

Both on irc.freenode.net

Conferences

  • EuroPython (Europe)
  • PyCon (US)
  • Other Python conferences around the world: Argentina, Asia Pacific, Australia, Brasil, Finland, France, India, Ireland, Italy, Japan, New Zealand, Poland, UK, Ukraine, ...
  • Other Python-related conferences: ScyPy conferences (Euro, US, India), DjangoCon (Euro, US), ...

Tools

  • Issue Tracker
  • buildbot
  • Mercurial (HG)

Issue Tracker

  • http://bugs.python.org/
  • Used to track bugs and feature requests
  • Based on Roundup
  • Supports Rietveld
  • Weekly issues reports sent to python-dev
  • There's also a meta-tracker

Issue Tracker Stats

http://bugs.python.org/issue?@template=stats

  • open 4591 (2160 with patches)
  • closed 29248
  • total 33839

Buildbots

Whenever someone pushes a change, they:

  • recompile Python
  • execute the whole test suite
  • report the results back

on different machines/OSes/architectures/python versions

Mercurial (HG)

HG

  • Terms and concepts
  • Repo structure
  • Status of the branches
  • Merging order
  • Basic usage
  • Clones organization

Terms and concepts

  • Repository (repo) The dir named .hg in the repository root dir that contains the history of the project.
  • Clone A copy of a repository.
  • Committing Saving local changes to the repository.
  • Updating Applying changes from the repo to the local copy.
  • Changeset (cset) An atomic collection of changes to files in a repository.
  • Pushing/Pulling Exchanging changesets from a repo to another.

Repo structure

7 named branches in the repo:

  • default
  • 3.4
  • 3.3
  • 3.2
  • 3.1
  • 2.7
  • 2.6

"default" will be the next 3.x

Status of the branches

development → bug-fix only → security-fix only

Release Status
3.5 (default) development
3.4 / 2.7 bug-fix only
3.3 / 2.6 security-fix only

Merging order

  • Two parallel lines of development

  • Changes applied to the oldest applicable branch first
    • merged with newer branches
    • copied/grafted from 2.x to 3.x or vice versa

(3.1 →) 3.2 → default

(2.6 →) 2.7

Basic usage

  • hg init creates a new repository
    • ...but you won't need this
  • hg clone creates a copy of an existing repo:
    • hg clone http://hg.python.org/cpython or
    • hg clone ssh://hg@hg.python.org/cpython (for committers)

  • hg branches to see the list of branches

  • hg branch to see the current branch

  • hg up <branchname> to change branch

Basic usage

  • hg pull to get the latest changesets in the local repo

  • hg up to apply them to the working copy
    • hg pull -u to do both at once

  • hg stat to see the changed file in the working copy

  • hg diff to see the changed code in the working copy

  • hg ci -m 'message' to save local changes in the repo

  • hg push to send the committed cset to another repo

Clones organization

Single clone (best for occasional contributors):

  • Pros: simpler configuration/setup
  • Cons: need to rebuild everything to work on another branch

Multiple clones (best for committers):

  • Pros: no branch changes, no recompilations
  • Cons: more complex configuration/setup

See https://docs.python.org/devguide/committing.html#clones-setup

Clones organization

  • You can work on several different bugs/features at once

  • Several ways to do it:
    • a clone per issue
    • a branch per issue
    • mq/shelve/evolve Mercurial extensions
    • plain old patches (recommended)
    • ...

Workflow

  1. Find something to fix/enhance
  2. Get a clone of Python
  3. Update and compile
  4. Fix/enhance it and check that it works
  5. Run the tests with regrtest
  6. Create or apply a patch
  7. Get someone else to review it

See https://docs.python.org/devguide/#quick-start

Find something to fix/enhance

  • A problem that you encountered
  • An issue in the tracker
  • A feature request
  • A broken test
  • A red buildbot
  • A new test to add
  • An "XXX" in the code
  • A problem in the documentation

Get a clone of Python

The main Python repo is at http://hg.python.org/cpython

read-only hg clone http://hg.python.org/cpython

read-write (for committers) hg clone ssh://hg@hg.python.org/cpython

Update and compile

Update:

  • hg pull -u to update the clone and the working copy

Compile:

  • ./configure --with-pydebug && make -j2 to compile Python
    • --with-pydebug to compile a debug build
    • -j2 to use 2 (or more) cores

You need to run make only if C files change.

Python supports incremental compilation.

Fix/enhance it and check it

For bugs:

  1. write the tests and check that they fail
  2. fix the bug (and possibly run make)
  3. run the tests and check that they pass

For new features the procedure is the same, but:

  • you might have to ask on python-ideas/python-dev first
  • you might have to write a PEP for major features/changes

Fix/enhance the documentation

  • The documentation is in Doc/

  • Written in ReST
    • easy to learn
    • easy to read/write
  • Built with Sphinx
    • use make html (in Doc/) to build the HTML doc and check the result in Docbuild/html/
    • make latex (in Doc/) and then make all-pdf (in Doc/build/latex/) to build the PDFs
    • supports incremental building

Run the tests with regrtest

Runner of the Python test suite. Lives in Lib/test/regrtest.py.

Examples:

  • ./python -m test to run the test suite
  • ./python -m test -uall to run all the tests
  • ./python -m test -j4 to run the tests in 4 processes
  • ./python -m test test_foo to run test_foo
  • ./python -m test -R 3:2 to check for refleaks
  • ./python -m test --help to see all the options

Use ./python -m test.regrtest on Python 2.

Create or apply a patch

Create a patch:

  • Run make patchcheck to check your changes.
  • Do hg diff > issue1234.diff

Apply it:

  • hg import --no-commit < issue4321.diff

You can also try patches uploaded on the tracker and:

  1. see if they apply cleanly and on what branches
  2. see if they solve the problem on your machine
  3. possibly fix and update them
  4. report your findings on the tracker

Get someone else to review it

  • Upload the patch or add a link to a remote hg repo on the tracker
  • Get reviews on the issue or on the integrated Rietveld instance
  • Ask someone on #python-dev to take a look
  • Ask on the core-mentorship ML
  • Ask on the python-dev ML
  • When it's committed someone might review the commit

Most likely someone will ask you to fix something in the patch

Guidelines and tips

Remember to:

  • Follow the PEP 8 for Python code and the PEP 7 for C code
  • Follow the conventions that already exist in the file(s)
  • Add/update tests and run them with regrtest
  • Check for refleaks with a debug build (if you touched the C code)
  • Add/update documentation, docstrings and comments
  • Fix one issue at time
  • Make patches againts the oldest branch (usually 2.7/3.4)
  • Make the patches easy to review

Developers will usually take care of merging and updating Misc/NEWS.

Structure of CPython

  • Python tree
  • Lib
  • Objects
  • Modules
  • Python, Include, Misc
  • Doc

Python tree

$ ls --group-directories-first
build   Misc    RISCOS        libpython2.7.a  python
Demo    Modules Tools         LICENSE         python-gdb.py
Doc     Objects config.log    Makefile        README
Grammar Parser  config.status Makefile.pre    setup.py
Include PC      configure     Makefile.pre.in
Lib     PCbuild configure.in  pyconfig.h
Mac     Python  install-sh    pyconfig.h.in

Lib

  • Contains most of the stdlib

  • Python modules and packages

  • Lib/test contains most of the Python test suite
    • some packages have their own test suite (e.g. Lib/ctypes/test)

Objects

Contains the C implementation of the Python objects:

  • lists (listobjects.c)
  • dicts (dictobject.c)
  • tuples (tupleobject.c)
  • strings (stringobject.c and unicodeobject.c)
  • sets (setobject.c)
  • files (fileobject.c)
  • etc...

Modules

Contains the C implementation of some Python modules:

  • math (mathmodule.c)
  • cmath (cmathmodule.c)
  • datetime (datetimemodule.c)
  • itertools (itertoolsmodule.c)
  • socket (socketmodule.c)
  • etc...

Python, Include, Misc

  • Python contains the core of CPython.

  • Include contains the headers (*.h) of the C files in Objects, Modules, and Python.

  • Misc contains other files, e.g.:
    • Misc/NEWS: where most of the changes are listed
    • Misc/ACKS: acknowledgments to contributors

Doc

Contains the ReST sources of the documentation.

  • Doc/library contains the ReST doc of the modules
  • Doc/c-api contains the ReST doc of the C API
  • Doc/build/html contains the HTML pages generated with make html

Questions

Questions?

Read the new devguide!