Becoming a Python Developer

1 Introduction

This article shows you how to become one of the people who develop Python, meaning not just a person who develops using Python, but a person who works on the core interpreter itself, and who contributes to the design of the language and the direction its development will take. This document will try to motivate you to take up Python development, explains the development process, and points you toward the online forums where discussion takes place.

2 Why Work on Python?

Why would you want to work on the Python core interpreter?

You're a Python user who wants to make Python more useful for your purposes.
You like hacking on sizable software systems in general, or language interpreters in particular.
You want to gain experience as a developer on an open source project.

Fairly often you can see postings in various online forums from programmers who want to get experience developing a free software project, asking what program they should write. Many programmers seem to go off and start their own new project, writing a new IRC client, mail program, or some similar application. However, it's more educational to join an existing and sizable project because by doing so, you can learn skills that aren't called into play for a smaller project. Python would be an excellent starting choice for the following reasons:

Because Python is a large project, there are lots of available tasks suitable for different temperaments and skill levels. If you enjoy writing prose and explaining things, you can write documentation. If you can program in Python (and it isn't difficult to learn), you can work on the library modules, tools, and demo programs that are included with the Python distribution. If you can program in C, you can write extension modules or hack on the interpreter itself. If you'd rather inspect someone else's code than write your own, you can examine submitted patches or fix bugs. You may think you have to be a highly skilled wizard who can rewrite the interpreter's main loop at the drop of a hat in order to contribute, but that's really not true; there are lots of things you can do, ranging from simple tasks to complex ones.
Python is over 10 years old, and its development process is quite mature at this point. There's a public CVS repository. There's a coding standard for both C and Python code. There's a framework for writing test suites. There's infrastructure for tracking bug reports and submitted patches. A newborn project will have to grow for some time before it needs such ancillary tools. Before a small project has acquired a group of developers, and many newborn projects never grow to that point, working on the project won't teach the use of such tools.
The most important skill Python can teach is the delicate skill of working in a diverse group. There's a core group of around 30 developers, roughly 10 of whom are very active and make the bulk of actual check-ins, and roughly 20 of whom make more occasional checkins and provide opinions and advice, but lots of people outside this core group contribute significantly, too. Developers oscillate between the more active and less active groups in an irregular way, depending on their free time, availability, and current interests; sometimes other things have to take precedence over Python hacking.
To work with this large and dispersed group, you'll have to learn who's the right person to ask each question, how to convince the other developers of the usefulness of a patch, how to offer helpful criticism, and how to take criticism.
C programmers will find that the C source code for the Python interpreter is clearly written and easy to dive into. It's relatively easy to familiarize yourself with the code, and to begin writing extensions or making changes.
Note, however, that it's possible to do a lot even if you don't know C at all. Working on the portions of the standard library that are written in Python is just as valuable for the project and is equally educational for you. Plus, programming in Python is much more pleasant than programming in C, because you get nicely formatted tracebacks instead of nasty core dumps and protection faults.

There are a few different implementations of Python. A quick overview of them follows.

The first implementation, and the one in widest use, is the implementation written in C. This is the implementation usually meant when people refer to ``Python'', but it's called CPython when people want to distinguish between it and some other implementation.
Stackless Python is a fork of CPython, but not one that diverges very far from the main tree. Its author, Christian Tismer, rewrote the main interpreter loop of CPython to minimize its use of the C stack; in particular, calling a Python function doesn't occupy any more room on the C stack. This means that, while CPython can only recurse a few thousand levels deep before filling up the C stack and crashing, Stackless can recurse to an unlimited depth. Stackless is also significantly faster than CPython (around 10%), supports continuations and lightweight threads, and has found a community of highly skilled users, who use it to do things such as writing massively-multiplayer online game. The Stackless Python home page is at http://www.stackless.com.
Jython is a reimplementation of Python, written in Java instead of C. (It was originally named JPython, but the name had to be changed for stupid trademark reasons.) Jython compiles Python code into Java bytecodes, and can seamlessly use any Java class directly from Python code, with no need to write an extension module first, as is necessary for CPython. The Jython home page is at http://www.jython.org.
Python for .NET is an experimental implementation of Python for the .NET Framework. Currently this seems to be a research effort, because while compiling Python to .NET bytecodes has been implemented, and the resulting code works, making the resulting code fast seems to be a difficult problem. See the Python.NET home page, at http://www.activestate.com/Initiatives/NET/Research.html, to get an overview of the current state of progress.

3 How Python is Developed

Guido van Rossum is the project's lead developer. In recognition of this role, he's sometimes jokingly called the Benevolent Dictator For Life, or BDFL; the acronym is occasionally used in python-dev postings, especially in contexts such as ``that will require a BDFL pronouncement''. In theory the BDFL makes all the decisions about what goes in to Python and what doesn't. In practice, often Guido will defer to someone else's expertise in a specialized domain; for example, Tim Peters is the resident master of floating point arcana, Jeremy Hylton usually wrestles the Python compiler, and so forth. Modules in the standard library are also often the responsibility of a particular individual who's the first choice to review patches or fix bugs in it, but anyone can modify any line of code at any time, and simple, obviously correct fixes can be applied by anyone.

3.1 PEPs

Because Python is a programming language, the approach taken to developing Python has to impose some rigidity and provide some resistance against accepting changes too easily. Outside parties have code implemented in Python, have written extension modules for it, and have embedded Python in their software, so it's important that the inconvenience of upgrading to new versions of Python is minimized. While a GUI application could have its innards completely rewritten with each new release as long as the user interface presented remained the same, that amount of source-level change would quickly drive Python's users away in frustation. Language changes can break old code, or might make the language too difficult for new users to learn.

As a way to ensure that changes are carefully considered, significant changes must be explained by a Python Enhancement Proposal, or PEP. PEPs are modelled on the Request For Comments documents used by the Internet Engineering Task Force, and describe a proposed change, including fairly complete documentation and a design rationale, and also record the community's consensus about a feature through the PEP's author taking note of people's comments and incorporating their feedback. PEPs are especially important if the suggested feature gets rejected, because the same ideas often come back again and again, resulting in lengthy discussion threads that always arrive at the same outcome. (Consider the number of times people that people have suggested adding a with statement, or support for interfaces.) Like an FAQ, which tries to reduce newsgroup traffic by answering questions before they're asked, PEPs try to reduce repeated suggestions.

All the PEPs are available online at http://www.python.org/peps/. PEP 1, ``PEP Purpose and Guidelines'' (online at http://www.python.org/peps/pep-0001.html), explains the purpose of PEPs, their life cycle, and the prescribed format for a PEP. Read it before beginning to write a PEP.

3.2 Design Principles

Each software project in the world has its own culture and style, its own approach to designing solutions to problems. Python has its own style, of course, and if you've learnt the language then that style has probably rubbed off on you to some extent. Python tries to keep things simple, to be orthogonal but not too much so, to assist the user as much as possible.

In June 1999, Tim Peters channeled Guido and listed 19 guiding principles for Python's design in a comp.lang.python posting. The principles shouldn't be taken too seriously, as they're not hard-and-fast constraints and for each rule you can probably list instances where it's been broken. Still, no one has had much disagreement with this list of design criteria.

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one -- and preferably only one -- obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than right now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Don't take these 19 aphorisms too seriously -- tattooing them on your body is probably a bad idea, for example -- but it's instructive to contemplate them. Some parallels can be drawn to the guiding principles of extreme programming, most notably the emphasis on ``Do the simplest thing that can possibly work''.

3.3 The CVS Tree

In May 2000, the Python CVS tree was moved to SourceForge (http://www.sf.net), and the bug tracking and patch submission tools offered by SourceForge were also adopted at that time. Using SourceForge's admittedly erratic project activity ranking, Python is often among the 10 most active projects.

Anyone can check out a read-only copy of the CVS tree by using anonymous CVS. First, you'll need to run the following command to anonymously log in to the CVS server:

cvs -d:pserver:anonymous@cvs.python.sourceforge.net:/cvsroot/python login

Then, you can check out the tree:

cvs -z3 \ 
  -d:pserver:anonymous@cvs.python.sourceforge.net:/cvsroot/python \
  co python

Running cvs -z3 update will update the CVS tree to the most recent version. Checkin messages and the accompanying diffs are sent to a mailing list, python-checkins. Anyone can subscribe by going to http://mail.python.org/mailman/listinfo/python-checkins.

Write access to the Python CVS tree is not automatically granted, though there's no formal process to go through to get it. If the python-dev team knows who you are, whether through mailing list discussion, having submitted patches, or some other interaction, then you can ask for full CVS access. You'll need to have a SourceForge ID, and provide it with your request.

3.4 Bug Tracking

To report a bug in Python, use the bug tracking tool at http://sourceforge.net/tracker/?group_id=5470&atid=105470. Don't report bugs by posting them to a mailing list, or sending them to a particular developer as a private e-mail; most likely the bug will end up buried under subsequent postings and e-mails, and stands a good chance of getting lost. Once a problem is recorded in the bug tracker, though, it's unlikely to get forgotten. It may sit unchanged for a while, but before releases someone will go over the outstanding bugs and fix or close them. PEP 3, ``Guidelines for Handling Bug Reports'' (online at http://www.python.org/peps/pep-0003.html), explains the procedures for handling bugs once they've been reported.

3.5 Patch Tracking

Just like bugs, and for much the same reasons, patches should be submitted to the patch tracker at http://sourceforge.net/tracker/?group_id=5470&atid=305470, and not through e-mail. Often a patch will need some modification before final acceptance, so be prepared to go through a few iterations before the patch is ready to be checked in.

Some conventions that should be followed when preparing a patch are:

Context diffs are preferred, so generate the patch using diff -c.
Patches to C code should follow Python's standard style, described in PEP 7, ``Style Guide for C Code'', at http://www.python.org/peps/pep-0007.html. If you're using Emacs to edit your C code, cc-mode supports Python's standard style; run the c-set-style command and select the 'python' style.
For patches to Jython, the standard formatting and indentation style for Java code should be followed.
Code written in Python should follow the style in Guido's style guide, described in PEP 8, ``Style Guide for Python Code'', (http://www.python.org/peps/pep-0008.html).

4 Mailing Lists

Python's development is coordinated through a variety of mailing lists and online forums, differing in focus and in the number of participants.

4.1 python-list

The most general and most high-traffic list is python-list (mail.python.org/mailman/listinfo/python-list), which is gatewayed to the Usenet newsgroup comp.lang.python. Pretty much anything Python-related is fair game for discussion, and the group is fairly tolerant of off-topic digressions. It's quite possible to have a reasoned discussion of programming languages other than Python; in the past there have been lengthy and informative threads about Lisp, floating point, and the principles of good software design, for example.

python-list mostly has discussion of development in Python, and not too much development of the Python interpreter itself, but most of the core developers still read the list, and those who can't keep up with the list can still make occasional appearances. Suggestions made on python-list frequently turn out to be frequently-made ones that are always rejected, but there have been other times when suggestions turned into a PEP, and then into a new feature.

Rudeness and personal attacks, even in reaction to blatant flamebait, are strongly frowned upon. People may strongly disagree on an issue, but usually discussion remains civil, though vehement. In case of an actual flamebait posting, you can ignore it, quietly plonk the offending poster in your killfile or mail filters, or write a sharp but still-polite response, but at all costs resist the urge to flame back. Generally python-list is a high-signal, low-noise group. The traffic is also quite high, currently running at around 150 posts per day.

4.2 python-announce

python-announce is a low-volume moderated list for posting Python-related announcements. New modules and programs are announced here, and, most relevant to Python developers, it's where PEPs are posted to get comments from the community. It's available through both the comp.lang.python.announce newsgroup, and by e-mail; the mailing list can be joined from http://mail.python.org/mailman/listinfo/python-announce-list.

4.3 Special Interest Groups

Special Interest Groups, or SIGs, are working groups focused on a single topic, such as database programming or text processing. A list of all SIGs is maintained at http://www.python.org/sigs/. SIGs are created by proposing a new SIG on the Meta-SIG mailing list (http://www.python.org/sigs/meta-sig/), writing a mission statement for it, and coming up with a list of deliverables. At least, that's the theory.

Some SIGs have been quite successful; the XML SIG has produced a distribution of Python XML software (http://pyxml.sourceforge.net), the Database SIG produced a standard Python API for accessing SQL databases (written up as PEPs 248 and 249), and the Distutils SIG produced the Distutils package included with Python 1.6. Other SIGs haven't been as productive, though. For example, the Catalog SIG and its predecessor, the Locator SIG, sputtered along for years without ever finalizing an implementation of a Python software catalog, while the Pattern SIG never stirred up much discussion about implementing software patterns in Python. Therefore, these days it's quite difficult to start a new SIG, and the maintainers of http://www.python.org will want some fairly clear evidence that the new SIG will attack enough interest (and developers) to achieve its goal. You might be happiest just creating a mailing list to discuss your topic, whether on a server you own or on Yahoo Groups (http://groups.yahoo.com) or some other free list service, rather than bother with the formalities of creating a SIG.

4.4 python-dev

Finally, python-dev is where the most focused activity takes place. Practically everyone with CVS write privileges is on python-dev. First drafts of PEPs are often posted here for initial review and rewriting before their more public appearance on python-announce.

python-dev is an invitation-only list, though the list address accepts e-mail from non-members, and the archives are public. Every two weeks, Michael Hudson writes summaries describing recent activity and discussions; summaries are archived at http://starship.python.net/crew/mwh/summaries/.

Sometimes, in response to a suggestion, people will post their votes on it, giving them as +1, -1, +0, or -0. This numbering comes from the voting scheme used by Apache: +1 indicates that the poster is in favor of the suggestion, -1 indicates they're against it, +0 indicates ``I don't care, but go ahead'', while -0 means ``I don't care, so why bother?''. In the Apache project, this voting is formalized and is actually how project decisions are made, but in Python, it's just a concise way to express opinions in a straw poll, and the result isn't binding in any way. The BDFL will take note of the reaction to a proposal, but is free to ignore it. While the BDFL can ignore community reaction, I can't think of an instance where he's actually done so in the face of united disapproval by the community. The closest case to that might be the print » statement, where everyone turned out to be divided 50/50 between liking it and hating it. Guido exercised his right to decide, and the feature was added to the language in Python 2.0. Some people still hate it; some people who argued against it, such as this author, have now grown sneakily fond of it.

5 Conclusion

The development process of any large software system continuously shifts and evolves to match its environment, so this document will probably never be either complete or totally accurate. Please report inaccuracies, errors, or suggestions for additional topics to me at akuchlin@mems-exchange.org. I hope that this guide convinces you that Python is a worthwhile project to spend time on, and give you enough information to begin contributing.

6 Acknowledgements

The author would like to thank the following people for offering suggestions on various drafts of this article: Aahz Maruch, Skip Montanaro, Guido van Rossum.

About this document ...

Becoming a Python Developer

This document was generated using the LaTeX2HTML translator.

LaTeX2HTML is Copyright © 1993, 1994, 1995, 1996, 1997, Nikos Drakos, Computer Based Learning Unit, University of Leeds, and Copyright © 1997, 1998, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The application of LaTeX2HTML to the Python documentation has been heavily tailored by Fred L. Drake, Jr. Original navigation icons were contributed by Christopher Petrilli.