PythonQAS

See http://cx-freeze.sourceforge.net/ for a distutils extension that allows you to create console and GUI executables from Python code. py2exe, the most popular extension for building Python 2.x-based executables, does not yet support Python 3 but a version that does is in development.

You need to do two things: the script file's mode must be executable and the first line must begin with #! followed by the path of the Python interpreter.

The first is done by executing chmod +x scriptfile or perhaps chmod 755 scriptfile.

The second can be done in a number of ways. The most straightforward way is to write

#!/usr/local/bin/python 

as the very first line of your file, using the pathname for where the Python interpreter is installed on your platform.

If you would like the script to be independent of where the Python interpreter lives, you can use the env program. Almost all Unix variants support the following, assuming the Python interpreter is in a directory on the user's PATH:

#!/usr/bin/env python 

Don't do this for CGI scripts. The PATH variable for CGI scripts is often very minimal, so you need to use the actual absolute pathname of the interpreter.

Occasionally, a user's environment is so full that the /usr/bin/env program fails; or there's no env program at all. In that case, you can try the following hack (due to Alex Rezinsky):

#! /bin/sh """:" exec python $0 ${1+"$@"} """ 

The minor disadvantage is that this defines the script's __doc__ string. However, you can fix that by adding

__doc__ = """...Whatever...""" 

To read or write complex binary data formats, it's best to use the struct module. It allows you to take a string containing binary data (usually numbers) and convert it to Python objects; and vice versa.

For example, the following code reads two 2-byte integers and one 4-byte integer in big-endian format from a file:

import struct with open(filename, "rb") as f: s = f.read(8) x, y, z = struct.unpack(">hhl", s) 

The '>' in the format string forces big-endian data; the letter 'h' reads one "short integer" (2 bytes), and 'l' reads one "long integer" (4 bytes) from the string.

For data that is more regular (e.g. a homogeneous list of ints or floats), you can also use the array module.

Use the standard library module smtplib.

Here's a very simple interactive mail sender that uses it. This method will work on any host that supports an SMTP listener.

import sys, smtplib fromaddr = input("From: ") toaddrs = input("To: ").split('','') print("Enter message, end with ^D:") msg = '''' while True: line = sys.stdin.readline() if not line: break msg += line # The actual mail send server = smtplib.SMTP(''localhost'') server.sendmail(fromaddr, toaddrs, msg) server.quit() 

A Unix-only alternative uses sendmail. The location of the sendmail program varies between systems; sometimes it is /usr/lib/sendmail, sometimes /usr/sbin/sendmail. The sendmail manual page will help you out. Here's some sample code:

import os SENDMAIL = "/usr/sbin/sendmail" # sendmail location p = os.popen("%s -t -i" % SENDMAIL, "w") p.write("To: receiver@example.com ") p.write("Subject: test ") p.write(" ") # blank line separating headers from body p.write("Some text ") p.write("some more text ") sts = p.close() if sts != 0: print("Sendmail exit status", sts) 

You probably tried to make a multidimensional array like this:

>>> A = [[None] * 2] * 3 

This looks correct if you print it:

>>> A [[None, None], [None, None], [None, None]] 

But when you assign a value, it shows up in multiple places:

>>> A[0][0] = 5 >>> A [[5, None], [5, None], [5, None]] 

The reason is that replicating a list with * doesn't create copies, it only creates references to the existing objects. The *3 creates a list containing 3 references to the same list of length two. Changes to one row will show in all rows, which is almost certainly not what you want.

The suggested approach is to create a list of the desired length first and then fill in each element with a newly created list:

A = [None] * 3 for i in range(3): A[i] = [None] * 2 

This generates a list containing 3 different lists of length two. You can also use a list comprehension:

w, h = 2, 3 A = [[None] * w for i in range(h)] 

Or, you can use an extension that provides a matrix datatype; Numeric Python is the best known.

Both static data and static methods (in the sense of C++ or Java) are supported in Python.

For static data, simply define a class attribute. To assign a new value to the attribute, you have to explicitly use the class name in the assignment:

class C: count = 0 # number of times C.__init__ called def __init__(self): C.count = C.count + 1 def getcount(self): return C.count # or return self.count 

c.count also refers to C.count for any c such that isinstance(c, C) holds, unless overridden by c itself or by some class on the base-class search path from c.__class__ back to C.

Caution: within a method of C, an assignment like self.count = 42 creates a new and unrelated instance named "count" in self's own dict. Rebinding of a class-static data name must always specify the class whether inside a method or not:

C.count = 314 

Static methods are possible:

class C: @staticmethod def static(arg1, arg2, arg3): # No ''self'' parameter! ... 

However, a far more straightforward way to get the effect of a static method is via a simple module-level function:

def getcount(): return C.count 

If your code is structured so as to define one class (or tightly related class hierarchy) per module, this supplies the desired encapsulation.

When a module is imported for the first time (or when the source file has changed since the current compiled file was created) a .pyc file containing the compiled code should be created in a __pycache__ subdirectory of the directory containing the .py file. The .pyc file will have a filename that starts with the same name as the .py file, and ends with .pyc, with a middle component that depends on the particular python binary that created it. (See PEP 3147 for details.)

One reason that a .pyc file may not be created is a permissions problem with the directory containing the source file, meaning that the __pycache__ subdirectory cannot be created. This can happen, for example, if you develop as one user but run as another, such as if you are testing with a web server.

Unless the PYTHONDONTWRITEBYTECODE environment variable is set, creation of a .pyc file is automatic if you're importing a module and Python has the ability (permissions, free space, etc...) to create a __pycache__ subdirectory and write the compiled module to that subdirectory.

Running Python on a top level script is not considered an import and no .pyc will be created. For example, if you have a top-level module foo.py that imports another module xyz.py, when you run foo (by typing python foo.py as a shell command), a .pyc will be created for xyz because xyz is imported, but no .pyc file will be created for foo since foo.py isn't being imported.

If you need to create a .pyc file for foo – that is, to create a .pyc file for a module that is not imported – you can, using the py_compile and compileall modules.

The py_compile module can manually compile any module. One way is to use the compile() function in that module interactively:

>>> import py_compile >>> py_compile.compile(''foo.py'') 

This will write the .pyc to a __pycache__ subdirectory in the same location as foo.py (or you can override that with the optional parameter cfile).

You can also automatically compile all files in a directory or directories using the compileall module. You can do it from the shell prompt by running compileall.py and providing the path of a directory containing Python files to compile:

python -m compileall . 

The pydoc module can create HTML from the doc strings in your Python source code. An alternative for creating API documentation purely from docstrings is epydoc. Sphinx can also include docstring content.

Yes, you can create built-in modules containing functions, variables, exceptions and even new types in C. This is explained in the document Extending and Embedding the Python Interpreter.

Most intermediate or advanced Python books will also cover this topic.

Yes, using the C compatibility features found in C++. Place extern "C" { ... } around the Python include files and put extern "C" before each function that is going to be called by the Python interpreter. Global or static C++ objects with constructors are probably not a good idea.

You have two choices: you can use nested scopes or you can use callable objects. For example, suppose you wanted to define linear(a,b) which returns a function f(x) that computes the value a*x+b. Using nested scopes:

def linear(a, b): def result(x): return a * x + b return result 

Or using a callable object:

class linear: def __init__(self, a, b): self.a, self.b = a, b def __call__(self, x): return self.a * x + self.b 

In both cases,

taxes = linear(0.3, 2) 

gives a callable object where taxes(10e6) == 0.3 * 10e6 + 2.

The callable object approach has the disadvantage that it is a bit slower and results in slightly longer code. However, note that a collection of callables can share their signature via inheritance:

class exponential(linear): # __init__ inherited def __call__(self, x): return self.a * (x ** self.b) 

Object can encapsulate state for several methods:

class counter: value = 0 def set(self, x): self.value = x def up(self): self.value = self.value + 1 def down(self): self.value = self.value - 1 count = counter() inc, dec, reset = count.up, count.down, count.set 

Here inc(), dec() and reset() act like functions which share the same counting variable.

Use a list:

["this", 1, "is", "an", "array"] 

Lists are equivalent to C or Pascal arrays in their time complexity; the primary difference is that a Python list can contain objects of many different types.

The array module also provides methods for creating arrays of fixed types with compact representations, but they are slower to index than lists. Also note that the Numeric extensions and others define array-like structures with various characteristics as well.

To get Lisp-style linked lists, you can emulate cons cells using tuples:

lisp_list = ("like", ("this", ("example", None) ) ) 

If mutability is desired, you could use lists instead of tuples. Here the analogue of lisp car is lisp_list[0] and the analogue of cdr is lisp_list[1]. Only do this if you're sure you really need to, because it's usually a lot slower than using Python lists.

On Windows, the standard Python installer already associates the .py extension with a file type (Python.File) and gives that file type an open command that runs the interpreter (D:\Program Files\Python\python.exe "%1" %*). This is enough to make scripts executable from the command prompt as 'foo.py'. If you'd rather be able to execute the script by simple typing 'foo' with no extension you need to add .py to the PATHEXT environment variable.

Here's a very brief summary of what started it all, written by Guido van Rossum:

I had extensive experience with implementing an interpreted language in the ABC group at CWI, and from working with this group I had learned a lot about language design. This is the origin of many Python features, including the use of indentation for statement grouping and the inclusion of very-high-level data types (although the details are all different in Python).

I had a number of gripes about the ABC language, but also liked many of its features. It was impossible to extend the ABC language (or its implementation) to remedy my complaints – in fact its lack of extensibility was one of its biggest problems. I had some experience with using Modula-2+ and talked with the designers of Modula-3 and read the Modula-3 report. Modula-3 is the origin of the syntax and semantics used for exceptions, and some other Python features.

I was working in the Amoeba distributed operating system group at CWI. We needed a better way to do system administration than by writing either C programs or Bourne shell scripts, since Amoeba had its own system call interface which wasn't easily accessible from the Bourne shell. My experience with error handling in Amoeba made me acutely aware of the importance of exceptions as a programming language feature.

It occurred to me that a scripting language with a syntax like ABC but with access to the Amoeba system calls would fill the need. I realized that it would be foolish to write an Amoeba-specific language, so I decided that I needed a language that was generally extensible.

During the 1989 Christmas holidays, I had a lot of time on my hand, so I decided to give it a try. During the next year, while still mostly working on it in my own time, Python was used in the Amoeba project with increasing success, and the feedback from colleagues made me add many early improvements.

In February 1991, after just over a year of development, I decided to post to USENET. The rest is in the Misc/HISTORY file.

Python versions are numbered A.B.C or A.B. A is the major version number – it is only incremented for really major changes in the language. B is the minor version number, incremented for less earth-shattering changes. C is the micro-level – it is incremented for each bugfix release. See PEP 6 for more information about bugfix releases.

Not all releases are bugfix releases. In the run-up to a new major release, a series of development releases are made, denoted as alpha, beta, or release candidate. Alphas are early releases in which interfaces aren't yet finalized; it's not unexpected to see an interface change between two alpha releases. Betas are more stable, preserving existing interfaces but possibly adding new modules, and release candidates are frozen, making no changes except as needed to fix critical bugs.

Alpha, beta and release candidate versions have an additional suffix. The suffix for an alpha version is "aN" for some small number N, the suffix for a beta version is "bN" for some small number N, and the suffix for a release candidate version is "cN" for some small number N. In other words, all versions labeled 2.0aN precede the versions labeled 2.0bN, which precede versions labeled 2.0cN, and those precede 2.0.

You may also find version numbers with a "+" suffix, e.g. "2.2+". These are unreleased versions, built directly from the Subversion trunk. In practice, after a final minor release is made, the Subversion trunk is incremented to the next minor version, which becomes the "a0" version, e.g. "2.4a0".

See also the documentation for sys.version, sys.hexversion, and sys.version_info.

The latest Python source distribution is always available from python.org, at https://www.python.org/downloads/. The latest development sources can be obtained via anonymous Mercurial access at https://hg.python.org/cpython.

The source distribution is a gzipped tar file containing the complete C source, Sphinx-formatted documentation, Python library modules, example programs, and several useful pieces of freely distributable software. The source will compile and run out of the box on most UNIX platforms.

Consult the Getting Started section of the Python Developer's Guide for more information on getting the source code and compiling it.

The standard documentation for the current stable version of Python is available at https://docs.python.org/3/. PDF, plain text, and downloadable HTML versions are also available at https://docs.python.org/3/download.html.

The documentation is written in reStructuredText and processed by the Sphinx documentation tool. The reStructuredText source for the documentation is part of the Python source distribution.

Alpha and beta releases are available from https://www.python.org/downloads/. All releases are announced on the comp.lang.python and comp.lang.python.announce newsgroups and on the Python home page at https://www.python.org/; an RSS feed of news is available.

You can also access the development version of Python through Mercurial. See https://docs.python.org/devguide/faq.html for details.

To report a bug or submit a patch, please use the Roundup installation at https://bugs.python.org/.

You must have a Roundup account to report bugs; this makes it possible for us to contact you if we have follow-up questions. It will also enable Roundup to send you updates as we act on your bug. If you had previously used SourceForge to report bugs to Python, you can obtain your Roundup password through Roundup's password reset procedure.

For more information on how Python is developed, consult the Python Developer's Guide.

Very stable. New, stable releases have been coming out roughly every 6 to 18 months since 1991, and this seems likely to continue. Currently there are usually around 18 months between major releases.

The developers issue "bugfix" releases of older versions, so the stability of existing releases gradually improves. Bugfix releases, indicated by a third component of the version number (e.g. 2.5.3, 2.6.2), are managed for stability; only fixes for known problems are included in a bugfix release, and it's guaranteed that interfaces will remain the same throughout a series of bugfix releases.

The latest stable releases can always be found on the Python download page. There are two recommended production-ready versions at this point in time, because at the moment there are two branches of stable releases: 2.x and 3.x. Python 3.x may be less useful than 2.x, since currently there is more third party software available for Python 2 than for Python 3. Python 2 code will generally not run unchanged in Python 3.

There are probably tens of thousands of users, though it's difficult to obtain an exact count.

Python is available for free download, so there are no sales figures, and it's available from many different sites and packaged with many Linux distributions, so download statistics don't tell the whole story either.

The comp.lang.python newsgroup is very active, but not all Python users post to the group or even read it.

The canonical way to share information across modules within a single program is to create a special module (often called config or cfg). Just import the config module in all modules of your application; the module then becomes available as a global name. Because there is only one instance of each module, any changes made to the module object get reflected everywhere. For example:

config.py:

x = 0 # Default value of the ''x'' configuration setting 

mod.py:

import config config.x = 1 

main.py:

import config import mod print(config.x) 

Note that using a module is also the basis for implementing the Singleton design pattern, for the same reason.

Collect the arguments using the * and ** specifiers in the function's parameter list; this gives you the positional arguments as a tuple and the keyword arguments as a dictionary. You can then pass these arguments when calling another function by using * and **:

def f(x, *args, **kwargs): ... kwargs[''width''] = ''14.3c'' ... g(x, *args, **kwargs) 

Remember that arguments are passed by assignment in Python. Since assignment just creates references to objects, there's no alias between an argument name in the caller and callee, and so no call-by-reference per se. You can achieve the desired effect in a number of ways.

1. By returning a tuple of the results:

   def func2(a, b): a = ''new-value'' # a and b are local names b = b + 1 # assigned to new objects return a, b # return new values x, y = ''old-value'', 99 x, y = func2(x, y) print(x, y) # output: new-value 100 

   This is almost always the clearest solution.

2. By using global variables. This isn't thread-safe, and is not recommended.

3. By passing a mutable (changeable in-place) object:

   def func1(a): a[0] = ''new-value'' # ''a'' references a mutable list a[1] = a[1] + 1 # changes a shared object args = [''old-value'', 99] func1(args) print(args[0], args[1]) # output: new-value 100 

4. By passing in a dictionary that gets mutated:

   def func3(args): args[''a''] = ''new-value'' # args is a mutable dictionary args[''b''] = args[''b''] + 1 # change it in-place args = {''a'': ''old-value'', ''b'': 99} func3(args) print(args[''a''], args[''b'']) 

5. Or bundle up values in a class instance:

   class callByRef: def __init__(self, **args): for (key, value) in args.items(): setattr(self, key, value) def func4(args): args.a = ''new-value'' # args is a mutable callByRef args.b = args.b + 1 # change object in-place args = callByRef(a=''old-value'', b=99) func4(args) print(args.a, args.b) 

   There's almost never a good reason to get this complicated.

Your best choice is to return a tuple containing the multiple results.

In general, try copy.copy() or copy.deepcopy() for the general case. Not all objects can be copied, but most can.

Some objects can be copied more easily. Dictionaries have a copy() method:

newdict = olddict.copy() 

Sequences can be copied by slicing:

new_l = l[:] 

For an instance x of a user-defined class, dir(x) returns an alphabetized list of the names containing the instance attributes and methods and attributes defined by its class.

Generally speaking, it can't, because objects don't really have names. Essentially, assignment always binds a name to a value; The same is true of def and class statements, but in that case the value is a callable. Consider the following code:

>>> class A: ...  pass ... >>> B = A >>> a = B() >>> b = a >>> print(b) <__main__.A object at 0x16D07CC> >>> print(a) <__main__.A object at 0x16D07CC> 

Arguably the class has a name: even though it is bound to two names and invoked through the name B the created instance is still reported as an instance of class A. However, it is impossible to say whether the instance's name is a or b, since both names are bound to the same value.

Generally speaking it should not be necessary for your code to "know the names" of particular values. Unless you are deliberately writing introspective programs, this is usually an indication that a change of approach might be beneficial.

In comp.lang.python, Fredrik Lundh once gave an excellent analogy in answer to this question:

The same way as you get the name of that cat you found on your porch: the cat (object) itself cannot tell you its name, and it doesn't really care – so the only way to find out what it's called is to ask all your neighbours (namespaces) if it's their cat (object)...

....and don't be surprised if you'll find that it's known by many names, or no name at all!

To specify an octal digit, precede the octal value with a zero, and then a lower or uppercase "o". For example, to set the variable "a" to the octal value "10" (8 in decimal), type:

>>> a = 0o10 >>> a 8 

Hexadecimal is just as easy. Simply precede the hexadecimal number with a zero, and then a lower or uppercase "x". Hexadecimal digits can be specified in lower or uppercase. For example, in the Python interpreter:

>>> a = 0xa5 >>> a 165 >>> b = 0XB2 >>> b 178 

For integers, use the built-in int() type constructor, e.g. int(''144'') == 144. Similarly, float() converts to floating-point, e.g. float(''144'') == 144.0.

By default, these interpret the number as decimal, so that int(''0144'') == 144 and int(''0x144'') raises ValueError. int(string, base) takes the base to convert from as a second optional argument, so int(''0x144'', 16) == 324. If the base is specified as 0, the number is interpreted using Python's rules: a leading '0o' indicates octal, and '0x' indicates a hex number.

Do not use the built-in function eval() if all you need is to convert strings to numbers. eval() will be significantly slower and it presents a security risk: someone could pass you a Python expression that might have unwanted side effects. For example, someone could pass __import__(''os'').system("rm -rf $HOME") which would erase your home directory.

eval() also has the effect of interpreting numbers as Python expressions, so that e.g. eval(''09'') gives a syntax error because Python does not allow leading '0' in a decimal number (except '0').

To convert, e.g., the number 144 to the string '144', use the built-in type constructor str(). If you want a hexadecimal or octal representation, use the built-in functions hex() or oct(). For fancy formatting, see the Format String Syntax section, e.g. "{:04d}".format(144) yields ''0144'' and "{:.3f}".format(1.0/3.0) yields ''0.333''.

You can't, because strings are immutable. In most situations, you should simply construct a new string from the various parts you want to assemble it from. However, if you need an object with the ability to modify in-place unicode data, try using an io.StringIO object or the array module:

>>> import io >>> s = "Hello, world" >>> sio = io.StringIO(s) >>> sio.getvalue() ''Hello, world'' >>> sio.seek(7) 7 >>> sio.write("there!") 6 >>> sio.getvalue() ''Hello, there!'' >>> import array >>> a = array.array(''u'', s) >>> print(a) array(''u'', ''Hello, world'') >>> a[0] = ''y'' >>> print(a) array(''u'', ''yello, world'') >>> a.tounicode() ''yello, world'' 

There are various techniques.

• The best is to use a dictionary that maps strings to functions. The primary advantage of this technique is that the strings do not need to match the names of the functions. This is also the primary technique used to emulate a case construct:

  def a(): pass def b(): pass dispatch = {''go'': a, ''stop'': b} # Note lack of parens for funcs dispatch[get_input()]() # Note trailing parens to call function 

• Use the built-in function getattr():

  import foo getattr(foo, ''bar'')() 

  Note that getattr() works on any object, including classes, class instances, modules, and so on.

  This is used in several places in the standard library, like this:

  class Foo: def do_foo(self): ... def do_bar(self): ... f = getattr(foo_instance, ''do_'' + opname) f() 

• Use locals() or eval() to resolve the function name:

  def myFunc(): print("hello") fname = "myFunc" f = locals()[fname] f() f = eval(fname) f() 

  Note: Using eval() is slow and dangerous. If you don't have absolute control over the contents of the string, someone could pass a string that resulted in an arbitrary function being executed.

That's a tough one, in general. First, here are a list of things to remember before diving further:

• Performance characteristics vary across Python implementations. This FAQ focusses on CPython.
• Behaviour can vary across operating systems, especially when talking about I/O or multi-threading.
• You should always find the hot spots in your program before attempting to optimize any code (see the profile module).
• Writing benchmark scripts will allow you to iterate quickly when searching for improvements (see the timeit module).
• It is highly recommended to have good code coverage (through unit testing or any other technique) before potentially introducing regressions hidden in sophisticated optimizations.

That being said, there are many tricks to speed up Python code. Here are some general principles which go a long way towards reaching acceptable performance levels:

• Making your algorithms faster (or changing to faster ones) can yield much larger benefits than trying to sprinkle micro-optimization tricks all over your code.
• Use the right data structures. Study documentation for the Built-in Types and the collections module.
• When the standard library provides a primitive for doing something, it is likely (although not guaranteed) to be faster than any alternative you may come up with. This is doubly true for primitives written in C, such as builtins and some extension types. For example, be sure to use either the list.sort() built-in method or the related sorted() function to do sorting (and see the Sorting HOW TO for examples of moderately advanced usage).
• Abstractions tend to create indirections and force the interpreter to work more. If the levels of indirection outweigh the amount of useful work done, your program will be slower. You should avoid excessive abstraction, especially under the form of tiny functions or methods (which are also often detrimental to readability).

If you have reached the limit of what pure Python can allow, there are tools to take you further away. For example, Cython can compile a slightly modified version of Python code into a C extension, and can be used on many different platforms. Cython can take advantage of compilation (and optional type annotations) to make your code significantly faster than when interpreted. If you are confident in your C programming skills, you can also write a C extension module yourself.

The type constructor tuple(seq) converts any sequence (actually, any iterable) into a tuple with the same items in the same order.

For example, tuple([1, 2, 3]) yields (1, 2, 3) and tuple(''abc'') yields (''a'', ''b'', ''c''). If the argument is a tuple, it does not make a copy but returns the same object, so it is cheap to call tuple() when you aren't sure that an object is already a tuple.

The type constructor list(seq) converts any sequence or iterable into a list with the same items in the same order. For example, list((1, 2, 3)) yields [1, 2, 3] and list(''abc'') yields [''a'', ''b'', ''c'']. If the argument is a list, it makes a copy just like seq[:] would.

Use the reversed() built-in function, which is new in Python 2.4:

for x in reversed(sequence): ... # do something with x ... 

This won't touch your original sequence, but build a new copy with reversed order to iterate over.

With Python 2.3, you can use an extended slice syntax:

for x in sequence[::-1]: ... # do something with x ... 

See the Python Cookbook for a long discussion of many ways to do this:

https://code.activestate.com/recipes/52560/

If you don't mind reordering the list, sort it and then scan from the end of the list, deleting duplicates as you go:

if mylist: mylist.sort() last = mylist[-1] for i in range(len(mylist)-2, -1, -1): if last == mylist[i]: del mylist[i] else: last = mylist[i] 

If all elements of the list may be used as set keys (i.e. they are all hashable) this is often faster

mylist = list(set(mylist)) 

This converts the list into a set, thereby removing duplicates, and then back into a list.

Use a list comprehension:

result = [obj.method() for obj in mylist] 

Use collections.OrderedDict.

Merge them into an iterator of tuples, sort the resulting list, and then pick out the element you want.

>>> list1 = ["what", "I''m", "sorting", "by"] >>> list2 = ["something", "else", "to", "sort"] >>> pairs = zip(list1, list2) >>> pairs = sorted(pairs) >>> pairs [("I''m", ''else''), (''by'', ''sort''), (''sorting'', ''to''), (''what'', ''something'')] >>> result = [x[1] for x in pairs] >>> result [''else'', ''sort'', ''to'', ''something''] 

An alternative for the last step is:

>>> result = [] >>> for p in pairs: result.append(p[1]) 

If you find this more legible, you might prefer to use this instead of the final list comprehension. However, it is almost twice as slow for long lists. Why? First, the append() operation has to reallocate memory, and while it uses some tricks to avoid doing that each time, it still has to do it occasionally, and that costs quite a bit. Second, the expression "result.append" requires an extra attribute lookup, and third, there's a speed reduction from having to make all those function calls.

Use the built-in function isinstance(obj, cls). You can check if an object is an instance of any of a number of classes by providing a tuple instead of a single class, e.g. isinstance(obj, (class1, class2, ...)), and can also check whether an object is one of Python's built-in types, e.g. isinstance(obj, str) or isinstance(obj, (int, float, complex)).

Note that most programs do not use isinstance() on user-defined classes very often. If you are developing the classes yourself, a more proper object-oriented style is to define methods on the classes that encapsulate a particular behaviour, instead of checking the object's class and doing a different thing based on what class it is. For example, if you have a function that does something:

def search(obj): if isinstance(obj, Mailbox): ... # code to search a mailbox elif isinstance(obj, Document): ... # code to search a document elif ... 

A better approach is to define a search() method on all the classes and just call it:

class Mailbox: def search(self): ... # code to search a mailbox class Document: def search(self): ... # code to search a document obj.search() 

Use the built-in super() function:

class Derived(Base): def meth(self): super(Derived, self).meth() 

For version prior to 3.0, you may be using classic classes: For a class definition such as class Derived(Base): ... you can call method meth() defined in Base (or one of Base's base classes) as Base.meth(self, arguments...). Here, Base.meth is an unbound method, so you need to provide the self argument.

You could define an alias for the base class, assign the real base class to it before your class definition, and use the alias throughout your class. Then all you have to change is the value assigned to the alias. Incidentally, this trick is also handy if you want to decide dynamically (e.g. depending on availability of resources) which base class to use. Example:

BaseAlias = <real base class> class Derived(BaseAlias): def meth(self): BaseAlias.meth(self) ... 

This answer actually applies to all methods, but the question usually comes up first in the context of constructors.

In C++ you'd write

class C { C() { cout << "No arguments "; } C(int i) { cout << "Argument is " << i << " "; } } 

In Python you have to write a single constructor that catches all cases using default arguments. For example:

class C: def __init__(self, i=None): if i is None: print("No arguments") else: print("Argument is", i) 

This is not entirely equivalent, but close enough in practice.

You could also try a variable-length argument list, e.g.

def __init__(self, *args): ... 

The same approach works for all method definitions.

Python does not keep track of all instances of a class (or of a built-in type). You can program the class's constructor to keep track of all instances by keeping a list of weak references to each instance.

A module can find out its own module name by looking at the predefined global variable __name__. If this has the value ''__main__'', the program is running as a script. Many modules that are usually used by importing them also provide a command-line interface or a self-test, and only execute this code after checking __name__:

def main(): print(''Running test...'') ... if __name__ == ''__main__'': main() 

Suppose you have the following modules:

foo.py:

from bar import bar_var foo_var = 1 

bar.py:

from foo import foo_var bar_var = 2 

The problem is that the interpreter will perform the following steps:

• main imports foo
• Empty globals for foo are created
• foo is compiled and starts executing
• foo imports bar
• Empty globals for bar are created
• bar is compiled and starts executing
• bar imports foo (which is a no-op since there already is a module named foo)
• bar.foo_var = foo.foo_var

The last step fails, because Python isn't done with interpreting foo yet and the global symbol dictionary for foo is still empty.

The same thing happens when you use import foo, and then try to access foo.foo_var in global code.

There are (at least) three possible workarounds for this problem.

Guido van Rossum recommends avoiding all uses of from <module> import ..., and placing all code inside functions. Initializations of global variables and class variables should use constants or built-in functions only. This means everything from an imported module is referenced as <module>.<name>.

Jim Roskind suggests performing steps in the following order in each module:

• exports (globals, functions, and classes that don't need imported base classes)
• import statements
• active code (including globals that are initialized from imported values).

van Rossum doesn't like this approach much because the imports appear in a strange place, but it does work.

Matthias Urlichs recommends restructuring your code so that the recursive import is not necessary in the first place.

These solutions are not mutually exclusive.

Consider using the convenience function import_module() from importlib instead:

z = importlib.import_module(''x.y.z'') 

A try/except block is extremely efficient if no exceptions are raised. Actually catching an exception is expensive. In versions of Python prior to 2.0 it was common to use this idiom:

try: value = mydict[key] except KeyError: mydict[key] = getvalue(key) value = mydict[key] 

This only made sense when you expected the dict to have the key almost all the time. If that wasn't the case, you coded it like this:

if key in mydict: value = mydict[key] else: value = mydict[key] = getvalue(key) 

For this specific case, you could also use value = dict.setdefault(key, getvalue(key)), but only if the getvalue() call is cheap enough because it is evaluated in all cases.

The details of Python memory management depend on the implementation. The standard implementation of Python, CPython, uses reference counting to detect inaccessible objects, and another mechanism to collect reference cycles, periodically executing a cycle detection algorithm which looks for inaccessible cycles and deletes the objects involved. The gc module provides functions to perform a garbage collection, obtain debugging statistics, and tune the collector's parameters.

Other implementations (such as Jython or PyPy), however, can rely on a different mechanism such as a full-blown garbage collector. This difference can cause some subtle porting problems if your Python code depends on the behavior of the reference counting implementation.

In some Python implementations, the following code (which is fine in CPython) will probably run out of file descriptors:

for file in very_long_list_of_files: f = open(file) c = f.read(1) 

Indeed, using CPython's reference counting and destructor scheme, each new assignment to f closes the previous file. With a traditional GC, however, those file objects will only get collected (and closed) at varying and possibly long intervals.

If you want to write code that will work with any Python implementation, you should explicitly close the file or use the with statement; this will work regardless of memory management scheme:

for file in very_long_list_of_files: with open(file) as f: c = f.read(1) 

Python's lists are really variable-length arrays, not Lisp-style linked lists. The implementation uses a contiguous array of references to other objects, and keeps a pointer to this array and the array's length in a list head structure.

This makes indexing a list a[i] an operation whose cost is independent of the size of the list or the value of the index.

When items are appended or inserted, the array of references is resized. Some cleverness is applied to improve the performance of appending items repeatedly; when the array must be grown, some extra space is allocated so the next few times don't require an actual resize.

Python's dictionaries are implemented as resizable hash tables. Compared to B-trees, this gives better performance for lookup (the most common operation by far) under most circumstances, and the implementation is simpler.

Dictionaries work by computing a hash code for each key stored in the dictionary using the hash() built-in function. The hash code varies widely depending on the key and a per-process seed; for example, "Python" could hash to -539294296 while "python", a string that differs by a single bit, could hash to 1142331976. The hash code is then used to calculate a location in an internal array where the value will be stored. Assuming that you're storing keys that all have different hash values, this means that dictionaries take constant time – O(1), in computer science notation – to retrieve a key. It also means that no sorted order of the keys is maintained, and traversing the array as the .keys() and .items() do will output the dictionary's content in some arbitrary jumbled order that can change with every invocation of a program.

An interface specification for a module as provided by languages such as C++ and Java describes the prototypes for the methods and functions of the module. Many feel that compile-time enforcement of interface specifications helps in the construction of large programs.

Python 2.6 adds an abc module that lets you define Abstract Base Classes (ABCs). You can then use isinstance() and issubclass() to check whether an instance or a class implements a particular ABC. The collections.abc module defines a set of useful ABCs such as Iterable, Container, and MutableMapping.

For Python, many of the advantages of interface specifications can be obtained by an appropriate test discipline for components. There is also a tool, PyChecker, which can be used to find problems due to subclassing.

A good test suite for a module can both provide a regression test and serve as a module interface specification and a set of examples. Many Python modules can be run as a script to provide a simple "self test." Even modules which use complex external interfaces can often be tested in isolation using trivial "stub" emulations of the external interface. The doctest and unittest modules or third-party test frameworks can be used to construct exhaustive test suites that exercise every line of code in a module.

An appropriate testing discipline can help build large complex applications in Python as well as having interface specifications would. In fact, it can be better because an interface specification cannot test certain properties of a program. For example, the append() method is expected to add new elements to the end of some internal list; an interface specification cannot test that your append() implementation will actually do this correctly, but it's trivial to check this property in a test suite.

Writing test suites is very helpful, and you might want to design your code with an eye to making it easily tested. One increasingly popular technique, test-directed development, calls for writing parts of the test suite first, before you write any of the actual code. Of course Python allows you to be sloppy and not write test cases at all.

Check the Library Reference to see if there's a relevant standard library module. (Eventually you'll learn what's in the standard library and will be able to skip this step.)

For third-party packages, search the Python Package Index or try Google or another Web search engine. Searching for "Python" plus a keyword or two for your topic of interest will usually find something helpful.

Python comes with two testing frameworks. The doctest module finds examples in the docstrings for a module and runs them, comparing the output with the expected output given in the docstring.

The unittest module is a fancier testing framework modelled on Java and Smalltalk testing frameworks.

To make testing easier, you should use good modular design in your program. Your program should have almost all functionality encapsulated in either functions or class methods – and this sometimes has the surprising and delightful effect of making the program run faster (because local variable accesses are faster than global accesses). Furthermore the program should avoid depending on mutating global variables, since this makes testing much more difficult to do.

The "global main logic" of your program may be as simple as

if __name__ == "__main__": main_logic() 

at the bottom of the main module of your program.

Once your program is organized as a tractable collection of functions and class behaviours you should write test functions that exercise the behaviours. A test suite that automates a sequence of tests can be associated with each module. This sounds like a lot of work, but since Python is so terse and flexible it's surprisingly easy. You can make coding much more pleasant and fun by writing your test functions in parallel with the "production code", since this makes it easy to find bugs and even design flaws earlier.

"Support modules" that are not intended to be the main module of a program may include a self-test of the module.

if __name__ == "__main__": self_test() 

Even programs that interact with complex external interfaces may be tested when the external interfaces are unavailable by using "fake" interfaces implemented in Python.

For Unix variants there are several solutions. It's straightforward to do this using curses, but curses is a fairly large module to learn.

Be sure to use the threading module and not the _thread module. The threading module builds convenient abstractions on top of the low-level primitives provided by the _thread module.

Aahz has a set of slides from his threading tutorial that are helpful; see http://www.pythoncraft.com/OSCON2001/.

The easiest way is to use the new concurrent.futures module, especially the ThreadPoolExecutor class.

Or, if you want fine control over the dispatching algorithm, you can write your own logic manually. Use the queue module to create a queue containing a list of jobs. The Queue class maintains a list of objects and has a .put(obj) method that adds items to the queue and a .get() method to return them. The class will take care of the locking necessary to ensure that each job is handed out exactly once.

Here's a trivial example:

import threading, queue, time # The worker thread gets jobs off the queue. When the queue is empty, it # assumes there will be no more work and exits. # (Realistically workers will run until terminated.) def worker(): print(''Running worker'') time.sleep(0.1) while True: try: arg = q.get(block=False) except queue.Empty: print(''Worker'', threading.currentThread(), end='' '') print(''queue empty'') break else: print(''Worker'', threading.currentThread(), end='' '') print(''running with argument'', arg) time.sleep(0.5) # Create queue q = queue.Queue() # Start a pool of 5 workers for i in range(5): t = threading.Thread(target=worker, name=''worker %i'' % (i+1)) t.start() # Begin adding work to the queue for i in range(50): q.put(i) # Give threads time to run print(''Main thread sleeping'') time.sleep(5) 

When run, this will produce the following output:

Running worker Running worker Running worker Running worker Running worker Main thread sleeping Worker <Thread(worker 1, started 130283832797456)> running with argument 0 Worker <Thread(worker 2, started 130283824404752)> running with argument 1 Worker <Thread(worker 3, started 130283816012048)> running with argument 2 Worker <Thread(worker 4, started 130283807619344)> running with argument 3 Worker <Thread(worker 5, started 130283799226640)> running with argument 4 Worker <Thread(worker 1, started 130283832797456)> running with argument 5 ... 

Consult the module's documentation for more details; the Queue class provides a featureful interface.

Use os.remove(filename) or os.unlink(filename); for documentation, see the os module. The two functions are identical; unlink() is simply the name of the Unix system call for this function.

To remove a directory, use os.rmdir(); use os.mkdir() to create one. os.makedirs(path) will create any intermediate directories in path that don't exist. os.removedirs(path) will remove intermediate directories as long as they're empty; if you want to delete an entire directory tree and its contents, use shutil.rmtree().

To rename a file, use os.rename(old_path, new_path).

To truncate a file, open it using f = open(filename, "rb+"), and use f.truncate(offset); offset defaults to the current seek position. There's also os.ftruncate(fd, offset) for files opened with os.open(), where fd is the file descriptor (a small integer).

The shutil module also contains a number of functions to work on files including copyfile(), copytree(), and rmtree().

The shutil module contains a copyfile() function. Note that on MacOS 9 it doesn't copy the resource fork and Finder info.

For Win32, POSIX (Linux, BSD, etc.), Jython:

http://pyserial.sourceforge.net

For Unix, see a Usenet post by Mitch Chapman:

https://groups.google.com/groups?selm=34A04430.CF9@ohioee.com

I would like to retrieve web pages that are the result of POSTing a form. Is there existing code that would let me do this easily?

Yes. Here's a simple example that uses urllib.request:

#!/usr/local/bin/python import urllib.request # build the query string qs = "First=Josephine&MI=Q&Last=Public" # connect and send the server a path req = urllib.request.urlopen(''http://www.some-server.out-there'' ''/cgi-bin/some-cgi-script'', data=qs) with req: msg, hdrs = req.read(), req.info() 

Note that in general for percent-encoded POST operations, query strings must be quoted using urllib.parse.urlencode(). For example, to send name=Guy Steele, Jr.:

>>> import urllib.parse >>> urllib.parse.urlencode({''name'': ''Guy Steele, Jr.''}) ''name=Guy+Steele%2C+Jr.'' 

The select module is commonly used to help with asynchronous I/O on sockets.

To prevent the TCP connect from blocking, you can set the socket to non-blocking mode. Then when you do the connect(), you will either connect immediately (unlikely) or get an exception that contains the error number as .errno. errno.EINPROGRESS indicates that the connection is in progress, but hasn't finished yet. Different OSes will return different values, so you're going to have to check what's returned on your system.

You can use the connect_ex() method to avoid creating an exception. It will just return the errno value. To poll, you can call connect_ex() again later – 0 or errno.EISCONN indicate that you're connected – or you can pass this socket to select to check if it's writable.

The pickle library module solves this in a very general way (though you still can't store things like open files, sockets or windows), and the shelve library module uses pickle and (g)dbm to create persistent mappings containing arbitrary Python objects.

The standard module random implements a random number generator. Usage is simple:

import random random.random() 

This returns a random floating point number in the range [0, 1).

There are also many other specialized generators in this module, such as:

• randrange(a, b) chooses an integer in the range [a, b).
• uniform(a, b) chooses a floating point number in the range [a, b).
• normalvariate(mean, sdev) samples the normal (Gaussian) distribution.

Some higher-level functions operate on sequences directly, such as:

• choice(S) chooses random element from a given sequence
• shuffle(L) shuffles a list in-place, i.e. permutes it randomly

There's also a Random class you can instantiate to create independent multiple random number generators.

The highest-level function to do this is PyRun_SimpleString() which takes a single string argument to be executed in the context of the module __main__ and returns 0 for success and -1 when an exception occurred (including SyntaxError). If you want more control, use PyRun_String(); see the source for PyRun_SimpleString() in Python/pythonrun.c.

Call the function PyRun_String() from the previous question with the start symbol Py_eval_input; it parses an expression, evaluates it and returns its value.

That depends on the object's type. If it's a tuple, PyTuple_Size() returns its length and PyTuple_GetItem() returns the item at a specified index. Lists have similar functions, PyListSize() and PyList_GetItem().

For bytes, PyBytes_Size() returns its length and PyBytes_AsStringAndSize() provides a pointer to its value and its length. Note that Python bytes objects may contain null bytes so C's strlen() should not be used.

To test the type of an object, first make sure it isn't NULL, and then use PyBytes_Check(), PyTuple_Check(), PyList_Check(), etc.

There is also a high-level API to Python objects which is provided by the so-called 'abstract' interface – read Include/abstract.h for further details. It allows interfacing with any kind of Python sequence using calls like PySequence_Length(), PySequence_GetItem(), etc. as well as many other useful protocols such as numbers (PyNumber_Index() et al.) and mappings in the PyMapping APIs.

You can't. Use PyTuple_Pack() instead.

The PyObject_CallMethod() function can be used to call an arbitrary method of an object. The parameters are the object, the name of the method to call, a format string like that used with Py_BuildValue(), and the argument values:

PyObject * PyObject_CallMethod(PyObject *object, const char *method_name, const char *arg_format, ...); 

This works for any object that has methods – whether built-in or user-defined. You are responsible for eventually Py_DECREF()'ing the return value.

To call, e.g., a file object's "seek" method with arguments 10, 0 (assuming the file object pointer is "f"):

res = PyObject_CallMethod(f, "seek", "(ii)", 10, 0); if (res == NULL) { ... an exception occurred ... } else { Py_DECREF(res); } 

Note that since PyObject_CallObject() always wants a tuple for the argument list, to call a function without arguments, pass "()" for the format, and to call a function with one argument, surround the argument in parentheses, e.g. "(i)".

In Python code, define an object that supports the write() method. Assign this object to sys.stdout and sys.stderr. Call print_error, or just allow the standard traceback mechanism to work. Then, the output will go wherever your write() method sends it.

The easiest way to do this is to use the io.StringIO class:

>>> import io, sys >>> sys.stdout = io.StringIO() >>> print(''foo'') >>> print(''hello world!'') >>> sys.stderr.write(sys.stdout.getvalue()) foo hello world! 

A custom object to do the same would look like this:

>>> import io, sys >>> class StdoutCatcher(io.TextIOBase): ... def __init__(self): ... self.data = [] ... def write(self, stuff): ... self.data.append(stuff) ... >>> import sys >>> sys.stdout = StdoutCatcher() >>> print(''foo'') >>> print(''hello world!'') >>> sys.stderr.write(''''.join(sys.stdout.data)) foo hello world! 

You can get a pointer to the module object as follows:

module = PyImport_ImportModule("<modulename>"); 

If the module hasn't been imported yet (i.e. it is not yet present in sys.modules), this initializes the module; otherwise it simply returns the value of sys.modules["<modulename>"]. Note that it doesn't enter the module into any namespace – it only ensures it has been initialized and is stored in sys.modules.

You can then access the module's attributes (i.e. any name defined in the module) as follows:

attr = PyObject_GetAttrString(module, "<attrname>"); 

Calling PyObject_SetAttrString() to assign to variables in the module also works.

Depending on your requirements, there are many approaches. To do this manually, begin by reading the "Extending and Embedding" document. Realize that for the Python run-time system, there isn't a whole lot of difference between C and C++ – so the strategy of building a new Python type around a C structure (pointer) type will also work for C++ objects.

For C++ libraries, see Writing C is hard; are there any alternatives?.

When using GDB with dynamically loaded extensions, you can't set a breakpoint in your extension until your extension is loaded.

In your .gdbinit file (or interactively), add the command:

br _PyImport_LoadDynamicModule 

Then, when you run GDB:

$ gdb /local/bin/python gdb) run myscript.py gdb) continue # repeat until your extension is loaded gdb) finish # so that your extension is loaded gdb) br myfunction.c:50 gdb) continue 

Sometimes you want to emulate the Python interactive interpreter's behavior, where it gives you a continuation prompt when the input is incomplete (e.g. you typed the start of an "if" statement or you didn't close your parentheses or triple string quotes), but it gives you a syntax error message immediately when the input is invalid.

In Python you can use the codeop module, which approximates the parser's behavior sufficiently. IDLE uses this, for example.

The easiest way to do it in C is to call PyRun_InteractiveLoop() (perhaps in a separate thread) and let the Python interpreter handle the input for you. You can also set the PyOS_ReadlineFunctionPointer() to point at your custom input function. See Modules/readline.c and Parser/myreadline.c for more hints.

However sometimes you have to run the embedded Python interpreter in the same thread as your rest application and you can't allow the PyRun_InteractiveLoop() to stop while waiting for user input. The one solution then is to call PyParser_ParseString() and test for e.error equal to E_EOF, which means the input is incomplete). Here's a sample code fragment, untested, inspired by code from Alex Farber:

#include <Python.h> #include <node.h> #include <errcode.h> #include <grammar.h> #include <parsetok.h> #include <compile.h> int testcomplete(char *code) /* code should end in */ /* return -1 for error, 0 for incomplete, 1 for complete */ { node *n; perrdetail e; n = PyParser_ParseString(code, &_PyParser_Grammar, Py_file_input, &e); if (n == NULL) { if (e.error == E_EOF) return 0; return -1; } PyNode_Free(n); return 1; } 

Another solution is trying to compile the received string with Py_CompileString(). If it compiles without errors, try to execute the returned code object by calling PyEval_EvalCode(). Otherwise save the input for later. If the compilation fails, find out if it's an error or just more input is required - by extracting the message string from the exception tuple and comparing it to the string "unexpected EOF while parsing". Here is a complete example using the GNU readline library (you may want to ignore SIGINT while calling readline()):

#include <stdio.h> #include <readline.h> #include <Python.h> #include <object.h> #include <compile.h> #include <eval.h> int main (int argc, char* argv[]) { int i, j, done = 0; /* lengths of line, code */ char ps1[] = ">>> "; char ps2[] = "... "; char *prompt = ps1; char *msg, *line, *code = NULL; PyObject *src, *glb, *loc; PyObject *exc, *val, *trb, *obj, *dum; Py_Initialize (); loc = PyDict_New (); glb = PyDict_New (); PyDict_SetItemString (glb, "__builtins__", PyEval_GetBuiltins ()); while (!done) { line = readline (prompt); if (NULL == line) /* Ctrl-D pressed */ { done = 1; } else { i = strlen (line); if (i > 0) add_history (line); /* save non-empty lines */ if (NULL == code) /* nothing in code yet */ j = 0; else j = strlen (code); code = realloc (code, i + j + 2); if (NULL == code) /* out of memory */ exit (1); if (0 == j) /* code was empty, so */ code[0] = ''''; /* keep strncat happy */ strncat (code, line, i); /* append line to code */ code[i + j] = '' ''; /* append '' '' to code */ code[i + j + 1] = ''''; src = Py_CompileString (code, "<stdin>", Py_single_input); if (NULL != src) /* compiled just fine - */ { if (ps1 == prompt || /* ">>> " or */ '' '' == code[i + j - 1]) /* "... " and double '' '' */ { /* so execute it */ dum = PyEval_EvalCode (src, glb, loc); Py_XDECREF (dum); Py_XDECREF (src); free (code); code = NULL; if (PyErr_Occurred ()) PyErr_Print (); prompt = ps1; } } /* syntax error or E_EOF? */ else if (PyErr_ExceptionMatches (PyExc_SyntaxError)) { PyErr_Fetch (&exc, &val, &trb); /* clears exception! */ if (PyArg_ParseTuple (val, "sO", &msg, &obj) && !strcmp (msg, "unexpected EOF while parsing")) /* E_EOF */ { Py_XDECREF (exc); Py_XDECREF (val); Py_XDECREF (trb); prompt = ps2; } else /* some other syntax error */ { PyErr_Restore (exc, val, trb); PyErr_Print (); free (code); code = NULL; prompt = ps1; } } else /* some non-syntax error */ { PyErr_Print (); free (code); code = NULL; prompt = ps1; } free (line); } } Py_XDECREF(glb); Py_XDECREF(loc); Py_Finalize(); exit(0); } 

To dynamically load g++ extension modules, you must recompile Python, relink it using g++ (change LINKCC in the Python Modules Makefile), and link your extension module using g++ (e.g., g++ -shared -o mymodule.so mymodule.o).

This is not necessarily a straightforward question. If you are already familiar with running programs from the Windows command line then everything will seem obvious; otherwise, you might need a little more guidance.

Unless you use some sort of integrated development environment, you will end up typing Windows commands into what is variously referred to as a "DOS window" or "Command prompt window". Usually you can create such a window from your Start menu; under Windows 7 the menu selection is Start ‣ Programs ‣ Accessories ‣ Command Prompt. You should be able to recognize when you have started such a window because you will see a Windows "command prompt", which usually looks like this:

C:\> 

The letter may be different, and there might be other things after it, so you might just as easily see something like:

D:\YourName\Projects\Python> 

depending on how your computer has been set up and what else you have recently done with it. Once you have started such a window, you are well on the way to running Python programs.

You need to realize that your Python scripts have to be processed by another program called the Python interpreter. The interpreter reads your script, compiles it into bytecodes, and then executes the bytecodes to run your program. So, how do you arrange for the interpreter to handle your Python?

First, you need to make sure that your command window recognises the word "python" as an instruction to start the interpreter. If you have opened a command window, you should try entering the command python and hitting return.:

C:\Users\YourName> python 

You should then see something like:

Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> 

You have started the interpreter in "interactive mode". That means you can enter Python statements or expressions interactively and have them executed or evaluated while you wait. This is one of Python's strongest features. Check it by entering a few expressions of your choice and seeing the results:

>>> print("Hello") Hello >>> "Hello" * 3 ''HelloHelloHello'' 

Many people use the interactive mode as a convenient yet highly programmable calculator. When you want to end your interactive Python session, hold the Ctrl key down while you enter a Z, then hit the "Enter" key to get back to your Windows command prompt.

You may also find that you have a Start-menu entry such as Start ‣ Programs ‣ Python 3.3 ‣ Python (command line) that results in you seeing the >>> prompt in a new window. If so, the window will disappear after you enter the Ctrl-Z character; Windows is running a single "python" command in the window, and closes it when you terminate the interpreter.

If the python command, instead of displaying the interpreter prompt >>>, gives you a message like:

''python'' is not recognized as an internal or external command, operable program or batch file. 

or:

Bad command or filename 

then you need to make sure that your computer knows where to find the Python interpreter. To do this you will have to modify a setting called PATH, which is a list of directories where Windows will look for programs.

You should arrange for Python's installation directory to be added to the PATH of every command window as it starts. If you installed Python fairly recently then the command

dir C:\py* 

will probably tell you where it is installed; the usual location is something like C:\Python33. Otherwise you will be reduced to a search of your whole disk ... use Tools ‣ Find or hit the Search button and look for "python.exe". Supposing you discover that Python is installed in the C:\Python33 directory (the default at the time of writing), you should make sure that entering the command

c:\Python33\python 

starts up the interpreter as above (and don't forget you'll need a "Ctrl-Z" and an "Enter" to get out of it). Once you have verified the directory, you can add it to the system path to make it easier to start Python by just running the python command. This is currently an option in the installer as of CPython 3.3.

More information about environment variables can be found on the Using Python on Windows page.

Embedding the Python interpreter in a Windows app can be summarized as follows:

1. Do _not_ build Python into your .exe file directly. On Windows, Python must be a DLL to handle importing modules that are themselves DLL's. (This is the first key undocumented fact.) Instead, link to pythonNN.dll; it is typically installed in C:\Windows\System. NN is the Python version, a number such as "33" for Python 3.3.

   You can link to Python in two different ways. Load-time linking means linking against pythonNN.lib, while run-time linking means linking against pythonNN.dll. (General note: pythonNN.lib is the so-called "import lib" corresponding to pythonNN.dll. It merely defines symbols for the linker.)

   Run-time linking greatly simplifies link options; everything happens at run time. Your code must load pythonNN.dll using the Windows LoadLibraryEx() routine. The code must also use access routines and data in pythonNN.dll (that is, Python's C API's) using pointers obtained by the Windows GetProcAddress() routine. Macros can make using these pointers transparent to any C code that calls routines in Python's C API.

   Borland note: convert pythonNN.lib to OMF format using Coff2Omf.exe first.

2. If you use SWIG, it is easy to create a Python "extension module" that will make the app's data and methods available to Python. SWIG will handle just about all the grungy details for you. The result is C code that you link into your .exe file (!) You do _not_ have to create a DLL file, and this also simplifies linking.

3. SWIG will create an init function (a C function) whose name depends on the name of the extension module. For example, if the name of the module is leo, the init function will be called initleo(). If you use SWIG shadow classes, as you should, the init function will be called initleoc(). This initializes a mostly hidden helper class used by the shadow class.

   The reason you can link the C code in step 2 into your .exe file is that calling the initialization function is equivalent to importing the module into Python! (This is the second key undocumented fact.)

4. In short, you can use the following code to initialize the Python interpreter with your extension module.

   #include "python.h" ... Py_Initialize(); // Initialize Python. initmyAppc(); // Initialize (import) the helper class. PyRun_SimpleString("import myApp"); // Import the shadow class. 

5. There are two problems with Python's C API which will become apparent if you use a compiler other than MSVC, the compiler used to build pythonNN.dll.

   Problem 1: The so-called "Very High Level" functions that take FILE * arguments will not work in a multi-compiler environment because each compiler's notion of a struct FILE will be different. From an implementation standpoint these are very _low_ level functions.

   Problem 2: SWIG generates the following code when generating wrappers to void functions:

   Py_INCREF(Py_None); _resultobj = Py_None; return _resultobj; 

   Alas, Py_None is a macro that expands to a reference to a complex data structure called _Py_NoneStruct inside pythonNN.dll. Again, this code will fail in a mult-compiler environment. Replace such code by:

   return Py_BuildValue(""); 

   It may be possible to use SWIG's %typemap command to make the change automatically, though I have not been able to get this to work (I'm a complete SWIG newbie).

6. Using a Python shell script to put up a Python interpreter window from inside your Windows app is not a good idea; the resulting window will be independent of your app's windowing system. Rather, you (or the wxPythonWindow class) should create a "native" interpreter window. It is easy to connect that window to the Python interpreter. You can redirect Python's i/o to _any_ object that supports read and write, so all you need is a Python object (defined in your extension module) that contains read() and write() methods.

The FAQ does not recommend using tabs, and the Python style guide, PEP 8, recommends 4 spaces for distributed Python code; this is also the Emacs python-mode default.

Under any editor, mixing tabs and spaces is a bad idea. MSVC is no different in this respect, and is easily configured to use spaces: Take Tools ‣ Options ‣ Tabs, and for file type "Default" set "Tab size" and "Indent size" to 4, and select the "Insert spaces" radio button.

If you suspect mixed tabs and spaces are causing problems in leading whitespace, run Python with the -t switch or run Tools/Scripts/tabnanny.py to check a directory tree in batch mode.

Use the msvcrt module. This is a standard Windows-specific extension module. It defines a function kbhit() which checks whether a keyboard hit is present, and getch() which gets one character without echoing it.

Prior to Python 2.7 and 3.2, to terminate a process, you can use ctypes:

import ctypes def kill(pid): """kill function for Win32""" kernel32 = ctypes.windll.kernel32 handle = kernel32.OpenProcess(1, 0, pid) return (0 != kernel32.TerminateProcess(handle, 0)) 

In 2.7 and 3.2, os.kill() is implemented similar to the above function, with the additional feature of being able to send Ctrl+C and Ctrl+Break to console subprocesses which are designed to handle those signals. See os.kill() for further details.

Sometimes, when you download the documentation package to a Windows machine using a web browser, the file extension of the saved file ends up being .EXE. This is a mistake; the extension should be .TGZ.

Simply rename the downloaded file to have the .TGZ extension, and WinZip will be able to handle it. (If your copy of WinZip doesn't, get a newer one from https://www.winzip.com.)

The technique, attributed to Randal Schwartz of the Perl community, sorts the elements of a list by a metric which maps each element to its "sort value". In Python, just use the key argument for the sort() method:

Isorted = L[:] Isorted.sort(key=lambda s: int(s[10:15])) 

The key argument is new in Python 2.4, for older versions this kind of sorting is quite simple to do with list comprehensions. To sort a list of strings by their uppercase values:

tmp1 = [(x.upper(), x) for x in L] # Schwartzian transform tmp1.sort() Usorted = [x[1] for x in tmp1] 

To sort by the integer value of a subfield extending from positions 10-15 in each string:

tmp2 = [(int(s[10:15]), s) for s in L] # Schwartzian transform tmp2.sort() Isorted = [x[1] for x in tmp2] 

For versions prior to 3.0, Isorted may also be computed by

def intfield(s): return int(s[10:15]) def Icmp(s1, s2): return cmp(intfield(s1), intfield(s2)) Isorted = L[:] Isorted.sort(Icmp) 

but since this method calls intfield() many times for each element of L, it is slower than the Schwartzian Transform.

Many people used to C or Perl complain that they want to use this C idiom:

while (line = readline(f)) { // do something with line } 

where in Python you're forced to write this:

while True: line = f.readline() if not line: break ... # do something with line 

The reason for not allowing assignment in Python expressions is a common, hard-to-find bug in those other languages, caused by this construct:

if (x = 0) { // error handling } else { // code that only works for nonzero x } 

The error is a simple typo: x = 0, which assigns 0 to the variable x, was written while the comparison x == 0 is certainly what was intended.

Many alternatives have been proposed. Most are hacks that save some typing but use arbitrary or cryptic syntax or keywords, and fail the simple criterion for language change proposals: it should intuitively suggest the proper meaning to a human reader who has not yet been introduced to the construct.

An interesting phenomenon is that most experienced Python programmers recognize the while True idiom and don't seem to be missing the assignment in expression construct much; it's only newcomers who express a strong desire to add this to the language.

There's an alternative way of spelling this that seems attractive but is generally less robust than the "while True" solution:

line = f.readline() while line: ... # do something with line... line = f.readline() 

The problem with this is that if you change your mind about exactly how you get the next line (e.g. you want to change it into sys.stdin.readline()) you have to remember to change two places in your program – the second occurrence is hidden at the bottom of the loop.

The best approach is to use iterators, making it possible to loop through objects using the for statement. For example, file objects support the iterator protocol, so you can write simply:

for line in f: ... # do something with line... 

Answer 1: Unfortunately, the interpreter pushes at least one C stack frame for each Python stack frame. Also, extensions can call back into Python at almost random moments. Therefore, a complete threads implementation requires thread support for C.

Answer 2: Fortunately, there is Stackless Python, which has a completely redesigned interpreter loop that avoids the C stack.

Python lambda expressions cannot contain statements because Python's syntactic framework can't handle statements nested inside expressions. However, in Python, this is not a serious problem. Unlike lambda forms in other languages, where they add functionality, Python lambdas are only a shorthand notation if you're too lazy to define a function.

Functions are already first class objects in Python, and can be declared in a local scope. Therefore the only advantage of using a lambda instead of a locally-defined function is that you don't need to invent a name for the function – but that's just a local variable to which the function object (which is exactly the same type of object that a lambda expression yields) is assigned!

Practical answer:

Cython and Pyrex compile a modified version of Python with optional annotations into C extensions. Weave makes it easy to intermingle Python and C code in various ways to increase performance. Nuitka is an up-and-coming compiler of Python into C++ code, aiming to support the full Python language.

Theoretical answer:

Not trivially. Python's high level data types, dynamic typing of objects and run-time invocation of the interpreter (using eval() or exec()) together mean that a naïvely "compiled" Python program would probably consist mostly of calls into the Python run-time system, even for seemingly simple operations like x+1.

Several projects described in the Python newsgroup or at past Python conferences have shown that this approach is feasible, although the speedups reached so far are only modest (e.g. 2x). Jython uses the same strategy for compiling to Java bytecode. (Jim Hugunin has demonstrated that in combination with whole-program analysis, speedups of 1000x are feasible for small demo programs. See the proceedings from the 1997 Python conference for more information.)

More precisely, they can't end with an odd number of backslashes: the unpaired backslash at the end escapes the closing quote character, leaving an unterminated string.

Raw strings were designed to ease creating input for processors (chiefly regular expression engines) that want to do their own backslash escape processing. Such processors consider an unmatched trailing backslash to be an error anyway, so raw strings disallow that. In return, they allow you to pass on the string quote character by escaping it with a backslash. These rules work well when r-strings are used for their intended purpose.

If you're trying to build Windows pathnames, note that all Windows system calls accept forward slashes too:

f = open("/mydir/file.txt") # works fine! 

If you're trying to build a pathname for a DOS command, try e.g. one of

dir = r"	his\is\my\dos\dir" "\" dir = r"	his\is\my\dos\dir\ "[:-1] dir = "\this\is\my\dos\dir\" 

The global interpreter lock (GIL) is often seen as a hindrance to Python's deployment on high-end multiprocessor server machines, because a multi-threaded Python program effectively only uses one CPU, due to the insistence that (almost) all Python code can only run while the GIL is held.

Back in the days of Python 1.5, Greg Stein actually implemented a comprehensive patch set (the "free threading" patches) that removed the GIL and replaced it with fine-grained locking. Adam Olsen recently did a similar experiment in his python-safethread project. Unfortunately, both experiments exhibited a sharp drop in single-thread performance (at least 30% slower), due to the amount of fine-grained locking necessary to compensate for the removal of the GIL.

This doesn't mean that you can't make good use of Python on multi-CPU machines! You just have to be creative with dividing the work up between multiple processes rather than multiple threads. The ProcessPoolExecutor class in the new concurrent.futures module provides an easy way of doing so; the multiprocessing module provides a lower-level API in case you want more control over dispatching of tasks.

Judicious use of C extensions will also help; if you use a C extension to perform a time-consuming task, the extension can release the GIL while the thread of execution is in the C code and allow other threads to get some work done. Some standard library modules such as zlib and hashlib already do this.

It has been suggested that the GIL should be a per-interpreter-state lock rather than truly global; interpreters then wouldn't be able to share objects. Unfortunately, this isn't likely to happen either. It would be a tremendous amount of work, because many object implementations currently have global state. For example, small integers and short strings are cached; these caches would have to be moved to the interpreter state. Other object types have their own free list; these free lists would have to be moved to the interpreter state. And so on.

And I doubt that it can even be done in finite time, because the same problem exists for 3rd party extensions. It is likely that 3rd party extensions are being written at a faster rate than you can convert them to store all their global state in the interpreter state.

And finally, once you have multiple interpreters not sharing any state, what have you gained over running each interpreter in a separate process?

os.read() is a low-level function which takes a file descriptor, a small integer representing the opened file. os.popen() creates a high-level file object, the same type returned by the built-in open() function. Thus, to read n bytes from a pipe p created with os.popen(), you need to use p.read(n).

Yes, you can inherit from built-in classes such as int, list, dict, etc.

The Boost Python Library (BPL, http://www.boost.org/libs/python/doc/index.html) provides a way of doing this from C++ (i.e. you can inherit from an extension class written in C++ using the BPL).

That depends on where Python came from.

If someone installed it deliberately, you can remove it without hurting anything. On Windows, use the Add/Remove Programs icon in the Control Panel.

If Python was installed by a third-party application, you can also remove it, but that application will no longer work. You should use that application's uninstaller rather than removing Python directly.

If Python came with your operating system, removing it is not recommended. If you remove it, whatever tools were written in Python will no longer run, and some of them might be important to you. Reinstalling the whole system would then be required to fix things again.