This book discusses methods of Programming in Python, with particular emphasis on Internet-related applications. Python has grown with the Internet, and programmers across the planet use Python for all sorts of Internet applications, from the simplest CGI scripts, to complex server programs, to extensions of the Python-based Grail browser.

This book will also be of interest to people who want to learn Python, but who aren't especially interested in developing Internet applications. Much of the book covers aspects of Python programming of general interest, useful in all application domains, and even the Internet-specific portions of this book provide compelling and accessible examples of techniques that can be used in many different applications. For example, the object-oriented techniques described in the chapters on graphical interfaces and generating HTML are useful for many complex programming tasks, and programmers with no particular interest in graphical interfaces or HTML generation may nevertheless find these discussions useful.

Python, of course, is not only for Internet applications -- companies large and small, hobbyists, and highly skilled professional programmers all around the world use Python in all sorts of domains: to control complex hardware, as a steering language for supercomputer computations, as a graphical user interface application builder, and as a code generation language, and just for fun, among other things.

Python Gets the Job Done

Python is a general purpose programming language. Python may also serve as a glue language connecting many separate software components in a simple and flexible manner, or as a steering language where high-level Python control modules guide low-level operations implemented by subroutine libraries effected in other languages. Python is also easy to learn and use, so it could also serve as an interface for naive users of advanced applications, or as a first programming language.

Behold some qualitative properties of Python:

--Easy: Python's simple syntax and elegant and clean semantics is easy for programmers and non-programmers to learn, read, and use. Experienced Programmers will be productive in Python in a day. Novices might be productive in Python in a day and a half. Python's basic syntax looks like a radical simplification of the algol/C/Pascal programming languages and as such is easily learned by people who have some experience with common mathematical notations and/or other common programming languages.

--Powerful: Almost any computational concept can be expressed briefly and directly in Python. Despite its "easiness" mentioned above, Python is not a toy language for novices only. Python includes powerful features such as:

--First-class functions and first-class everything else. Functions, methods, modules, classes, and other components of Python programs may be passed to functions or stored in data structures.

--Object orientation with multiple inheritance and late binding. Python allows creation of object-oriented class hierarchies, and the object referred to by name.attribute is determined at run time via a dynamic name search. For example, as demonstrated in the chapter on generating HTML, these features allow straightforward expression of complex computational concepts, and a high degree of code-reuse.

--Object-oriented and named exception handling. Errors or other exceptional conditions in Python programs can be trapped using try..except statements, and finalization actions can be specified using try..finally. This greatly simplifies code that may encounter exceptional conditions that interrupt the normal flow of control of the program. Exceptions may also be organized into inheritance structures.

--Dynamic calling sequences. Python callable objects can accept optional arguments, keyword arguments, or unlimited numbers of arguments. These features allow very generic and highly configurable operations to be implemented in terse Python declarations.

Highly skilled programmers will find that even the most subtle idea usually translates to a short and clear segment of Python code using the expressive power available. Large complex programs may be implemented in Python, and the end result may be working much sooner than expected, in a smaller and less complex form than anticipated.

--Extensible and Flexible: Python can be extended easily to interface with other software systems, and Python can also be easily incorporated into other programs as a component. Furthermore, Python allows extreme flexibility in the treatment of language components. For example, a Python module that is meant to interact with an external system can be tested using a briefly specified 'imitation' of the external system written in Python.

Here are some notable technical characteristics of the Python language that justify some of the qualitative claims given above.

--Interpreted: Python is dynamically interpreted language that supports byte compilation. Python programs may be developed and tested with the help of the interactive mode of the Python interpreter which allows program components to be debugged, traced, profiled, and tested interactively. Python byte code is also machine independent and may be executed on different hardware and software platforms without recompilation. For example, advanced network applications could load python byte code from a remote machine and execute the code dynamically.

--Object Oriented: Python supports object-oriented class structures with multiple inheritance and late binding. Python is also truly object oriented to the core: Everything in Python is an object organized in a flexible and general internal object framework that is beautiful to behold. Experienced C programmers who delve into the source code of the Python distribution will find the core implementation a delightful read. Should the need arise, programmers can create alternate implementations for any basic component of the Python language (such as numbers or even object classes) either in Python code implementations or via compiled language extensions. For example, the numerical Python extensions add general arbitrary dimensional matrices as a "number type," the MESS extensions define an alternate object model for Python, and the ni.py module defines hierarchical module structures -- all of these optional modules define alternative interpretations for basic operations of the Python core.

--Dynamic: Python uses dynamic typing and dynamic resolution of names. For example, an expression output.write(data) may write a data string to a file object output, or it may archive an arbitrary data object in an output object implemented in Python and stored in main memory, or it may do both at different times depending on the current binding of data and output. Thus a Python function which reads from input, transforming the read data and writing the result to output

def send_transformed(input, transform, output):
    while 1:
       data = input.read()
       if not data: return
       output.write( transform(data) )

need not know whether the input or output objects are files or other structures and what the read, write, or transform operations do. This flexibility allows expression of generic operations such as send_transformed that may be easily used in many different contexts.

Dynamic typing and dynamic name resolution when properly used, can greatly reduce the size of programs and greatly simplify testing and debugging, especially for programs that interact with complex external systems, some of which may not have been implemented yet.

--Orthogonally structured: Python is constructed from a small number of powerful constructs. The simplicity of the language allows different programmers to understand and use each other's code more easily, compared to more complex languages. Python's simplicity also allows new Python programmers to learn the language and become productive quickly.

At the same time the simplicity of the language does not sacrifice expressive power -- we claim that there are few features of other programming languages that cannot be emulated directly and easily using Python.

--Extendible: Python can be easily extended with interfaces to external programming libraries or new data types via compiled extension modules. Adding new compiled components to Python is easy -- much easier than writing stand-alone compiled programs. On most major platforms new compiled components may also be loaded into the interpreter dynamically, on demand.

--Embeddable: To another program the Python interpreter looks like a very simple applications programmer interface (API), and this book will show that the interpreter can be embedded within another application as a general purpose scripting/extension or glue/steering tool in a surprisingly straightforward manner.

--Stable, tested, and upwardly compatible: Python always has been upwardly compatible through the years and will continue to be upwardly compatible. New versions of the interpreter will always run programs written for old versions of the interpreter. At this point new versions of the interpreter include precious few bug fixes, because the Python core has been thoroughly tested and debugged in thousands of applications for the last several years. Most bugs found these days are obscure and avoidable.

--Portable and friendly to external software: Python is written in standard C using Posix input/output conventions and ports to all major platforms that support a Posix interface without problems. Have you ever heard of the BeBox? If you have, congratulations, you are among the select few. We hadn't heard of it until someone informed us that they had ported Python to the BeBox with few problems, just as Python ports to hundreds of other computing environments with little or no problems. Furthermore, internal aspects of the Python interpreter design -- such as its reference counting memory management scheme -- make it live peacefully with most other software components without technical difficulties.

--Freely available with unrestricted redistribution in source form: The Python copyright essentially protects the authors from legal jeopardy and prevents malicious users from attempting to hijack the copyright. Aside from that, Python programmers and users may use Python in source or binary form just about anyway they please. In particular, programmers may create products that use Python and release the product in binary-only form with all Python modules in byte-compiled-only form, and they may sell or give away the result in any manner they think will make them the wealthiest, or the most famous.

If Python is completely free, how can it be as good as we claim? Well, the real answer to that question is that if Python hadn't been free, it wouldn't have had hundreds of programmers testing it, offering suggestions, proposing bug-fixes and enhancements, and contributing extensions and libraries for several years now -- and it might never have gotten as good as it is. You can spend a lot of money to buy software that isn't as stable and generally useful as Python.

What Python Does Not Do Well (by Itself)

Python is not intended to be the perfect language for all purposes. We would suppose that most programs written today could be addressed well using Python, but Python alone (without special purpose extensions) would not be suitable for many applications that are generally better addressed by compiled components.

For example, Python alone might not be appropriate for the following example applications for the following reasons:

Data Compression Algorithms

A data compression algorithm translates a stream of data into a smaller form. Usually this involves examining each byte of the data, collecting statistical information on the bytes and their relative order, and emitting a compressed representation, one byte at a time. For suitably large data sizes, most data compressors implemented directly in Python are likely to be too slow for practical purposes due to the inherent overhead of the Python interpreter. To accomplish data compression or similar byte-by-byte applications for large data sets, either add an interface to a compiled special purpose library to Python or consider using a separate stand-alone compiled application, or some other approach.

Device Drivers

A device driver is essentially a piece of software added into an operating system, an interpreted language such as Python is inappropriate as part of an operating system kernel due to the overhead of the interpreter and its extensive use of dynamic memory allocation.

Applications with Millions of Ad Hoc Floating Point Operations

This case is essentially similar to the data compression example. An application that must perform millions of nonuniform floating point operations will likely be too slow in Python alone compared to using a compiled implementation. Critical calculations that are known to slow down a Python implementation could be implemented as C extensions to Python, to allow the higher level logic to be controlled by Python, if this is useful.

NOTE: If the operations are uniform (not ad hoc), then the situation is quite different. In this case consider using the numerical Python matrix extension to perform the calculations as aggregate operations. The result may actually be faster than a brute force compiled implementation, because the Python matrix extensions are highly efficient and convenient -- at least that's what physicists at the Lawrence Livermore National Labs tell us.

Mission Critical Database Operations

Many database-style operations may be performed directly and easily using Python programs, but for sufficiently important database transactions it is important to have reliable logging, concurrency control, and failure recovery. Implementation of these standard operations shouldn't be attempted by most programmers directly in any language. Instead, get a good database engine that implements these features, and use it.

NOTE: To use Python in conjunction with the database engine, add an extension module to Python that includes an API for accessing the database, or possibly use the OLE or CORBA or ILU remote object protocols, if this is appropriate.

Huge Monolithic Single Process Applications

Python can solve complex problems, often in simple ways that are extremely difficult to emulate using other languages. But some types of applications may not be greatly simplified by Python's powerful features, requiring very large amounts of very complex code all running in the same process. One possible problem with using Python alone in an application of this kind is that a poorly tested or poorly designed piece of code may corrupt a data structure that is internal to another object or module, because Python does not include data hiding features at this time. In this case it is worth considering using Python initially to prototype the application, and possibly later use Python as a component of the application once critical components have been implemented using C++ or some other language that directly enforces data hiding and strong type checking at compile time.

NOTE: There is little direct support for data hiding within Python itself, but extensions and embeddings of Python can provide rock solid interfaces that expose only permitted foreign operations to the Python interpreter. Python's restricted execution mode may also provide some (usually extreme) protection within the interpreter itself.

Highly Specialized Simple Operations

For example, if you want to scan a file for a pattern, it may be more appropriate to use a filter program such as the UNIX fgrep, egrep, or awk filters, rather than write a small Python program to the same end.

WARNING: Larger applications that must be maintained by multiple programmers might be more easily developed, tested, and modified using a flexible general purpose tool like Python, in place of specialized tools. Problems that seem simple and specialized often grow in complexity as they evolve, and as the complexity of the problem grows, the implementation might benefit from Python's advanced object model and error handling mechanisms -- features that are frequently difficult or impossible to emulate using specialized tools.

There are other types of applications that are not suitable for direct Python implementations without any support from compiled extension modules. Many of these applications might benefit from using Python as a component of the application, but others might be simply unsuitable for Python -- the experienced software engineer must make the call.

Most programming problems do not fall into any of these categories, they are not extremely simple and they are not hugely complex; they are not part of the operating system and they do not manipulate huge quantities of tiny bits of data. Most programs can be implemented using Python, often much more quickly than with other methods, and with more satisfactory results.

Python and the Internet

The Internet and intranets are probably the most popular applications areas for Python at the moment (although engineering applications are now close behind). Python is particularly well suited for Internet and intranet applications because these applications are often highly dynamic, somewhat complex, and often require interfaces with external systems -- Python's dynamism, its advanced features, and its ease of extension address each of these requirements in turn, and this book explains how.

The first six chapters are concerned with explaining the Python language, including basic and advanced features.

Introduction: This chapter introduces the book. You are reading it now.

Birds Eye Python: This chapter introduces some of the primary syntactic and semantic features of Python, and discusses the various modes of execution for the Python interpreter.

Playing: This chapter gently introduces the basic components of a Python program using interactive "play" with the Python interpreter. Much of the chapter discusses character string manipulations in detail, because this provides a very accessible platform for illustrating language features, and because strings are important in many application domains, especially in network programming. Other sections introduce the list type, the dictionary type and the use of Python's object-oriented features.

Intrinsic operations for common types: This chapter summarizes most semantic features of the Python language in a single place, by discussing the intrinsic operations of basic object types. Some readers may prefer to scan this chapter initially, rather than read it in detail -- referring back to appropriate sections of the chapter as the need arises.

Syntax and control: This chapter covers detailed issues of syntax and the semantics of Python control constructs. Many of these concepts are illustrated by example in the previous two chapters. The contents of this chapter, like those of the previous one, need not be understood in entirety before using the Python language, but the detailed information in this chapter may be useful later.

More goodies: A number of issues of Python programming that don't fit into the other categories (such as byte compilation, and documentation strings) fall into this chapter. Some discussions of this chapter are fairly advanced. Readers should read this chapter lightly at first, and they should not be too concerned if some of the content seems difficult at first, because they may never need to use the aspect they find difficult.

The following three chapters discuss Internet-specific programming using Python.

Formatting HTML, a case study in dynamic objects: This chapter describes techniques for formatting data into HTML representations for presentation on the Web, with particular emphasis on formatting the complex TABLE construct. This chapter intentionally makes use of Python's advanced dynamic features in order to simplify the implementation and to illustrate those features at the same time.

CGI programming: This chapter describes the CGI Mechanism used by the standard "Web" protocol HTTP to allow clients to submit form data (or other data) to Web site server machines, and to receive responses from the servers. The chapter also describes methods for implementing CGI programs using Python.

Protocols: This chapter discusses general issues of implementing protocols using Python, both via direct interactions with network socket structures and via the Python support libraries.

The remaining three chapters of the book describe other aspects of Python programming that are useful both in Internet programming and in other domains.

Extending Python: This chapter describes how to add new compiled components to the Python interpreter, either statically or via dynamically loaded libraries. The detailed example in this chapter describes how to add a new type to Python, and various ways the new module may be built and connected to the interpreter.

Embedding Python: This chapter describes how Python may be added as a component of another program. The detailed example in this chapter embeds Python under a Netscape Server program using the standard NSAPI applications programmer interface. The resulting embedding allows Python to act as a generic scripting and glue language for the Server program.

GUI Programming with Python: This chapter introduces the use of Python as a Graphical User Interface (GUI) language. This topic is quite deep and could fill another book because graphics libraries are complex even when interfaced to Python. This chapter describes the general properties of most GUI programs and gives a detailed description of the use of one of Python's more popular GUI interfaces, the Wpy package, which is portable between Microsoft and Unix platforms.

Background

Guido van Rossum, one of the present authors, wrote almost all of the core implementation of Python, and he continues to be the ultimate arbiter of the core language and its standard implementation. Python also includes numerous libraries and extension modules contributed by many programmers around the world, and many other extensions and contributed libraries are available from the http://www.python.org Python Software Activity (PSA) Web site. The PSA Web is also the standard location for the latest version of the Python interpreter source code, often with binary releases for many platforms available too.

Python was initially written as a component of the Amoeba operating system at CWI in the Netherlands beginning in 1989. Guido specifically designed the Python core architecture to allow uniform interfaces to external software in order to allow convenient high level control of arbitrarily many and arbitrarily complex components, and these features were immediately recognized and appreciated by hundreds of programmers on the Internet when Guido released Python for free distribution in 1991. Since then the community of Python users and programmers has ballooned, but its exact size is unknowable because it has passed from one machine to another silently over the years. Because many popular Internet services use Python (some of them very extensively, such as http://www.infoseek.com which boasts of billions of "hits") it is reasonable to suppose that hundreds of thousands of people have used Python programs in one way or another. Python is also used heavily in various noninternet applications at places such as NASA Mission Control, Lawrence Livermore National Labs, and the National Bureau of Standards.

Guido currently works at the Corporation for National Research Initiatives (CNRI), a nonprofit research institute that hosts the Python Software Activity (PSA) -- a clearing house for Python software and related information. Please see the PSA Web site for extensive further information on the Python universe.

Fasten Your Seatbelt, and Have Fun

We almost forgot to mention that programming using Python is a joy. New programmers will find using Python a blast, and experienced programmers will find Python refreshing and fun.

Once the Python interpreter is installed on your system, fire it up and play with it often. If the interpreter is named python, a very simple interaction with the interpreter might look like this.

% python
Python 1.3 (Dec  7 1995)  [GCC 2.6.3]
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> print "Hello world"
Hello world
>>> 2+8*3
26
>>> raise SystemExit
%

NOTE: Using raise SystemExit (without catching the exception) is a generic way to get out of the interpreter, but the end-of-file character (which is control-D on UNIX or control-Z on Microsoft platforms) also terminates the interactive interpreter. Be sure to enjoy yourself, and we hope you have as much fun as we have.