\chapter{Extending Python with \C{} or \Cpp{} \label{intro}}  It is quite easy to add new built-in modules to Python, if you knowhow to program in C.  Such \dfn{extension modules} can do two thingsthat can't be done directly in Python: they can implement new built-inobject types, and they can call C library functions and system calls. To support extensions, the Python API (Application ProgrammersInterface) defines a set of functions, macros and variables thatprovide access to most aspects of the Python run-time system.  ThePython API is incorporated in a C source file by including the header\code{"Python.h"}. The compilation of an extension module depends on its intended use aswell as on your system setup; details are given in later chapters.  \section{A Simple Example         \label{simpleExample}} Let's create an extension module called \samp{spam} (the favorite foodof Monty Python fans...) and let's say we want to create a Pythoninterface to the C library function \cfunction{system()}.\footnote{Aninterface for this function already exists in the standard module\module{os} --- it was chosen as a simple and straightforward example.}This function takes a null-terminated character string as argument andreturns an integer.  We want this function to be callable from Pythonas follows: \begin{verbatim}>>> import spam>>> status = spam.system("ls -l")\end{verbatim} Begin by creating a file \file{spammodule.c}.  (Historically, if amodule is called \samp{spam}, the C file containing its implementationis called \file{spammodule.c}; if the module name is very long, like\samp{spammify}, the module name can be just \file{spammify.c}.) The first line of our file can be: \begin{verbatim}#include <Python.h>\end{verbatim} which pulls in the Python API (you can add a comment describing thepurpose of the module and a copyright notice if you like). \begin{notice}[warning]  Since Python may define some pre-processor definitions which affect  the standard headers on some systems, you \emph{must} include  \file{Python.h} before any standard headers are included.\end{notice} All user-visible symbols defined by \file{Python.h} have a prefix of\samp{Py} or \samp{PY}, except those defined in standard header files.For convenience, and since they are used extensively by the Pythoninterpreter, \code{"Python.h"} includes a few standard header files:\code{<stdio.h>}, \code{<string.h>}, \code{<errno.h>}, and\code{<stdlib.h>}.  If the latter header file does not exist on yoursystem, it declares the functions \cfunction{malloc()},\cfunction{free()} and \cfunction{realloc()} directly. The next thing we add to our module file is the C function that willbe called when the Python expression \samp{spam.system(\var{string})}is evaluated (we'll see shortly how it ends up being called): \begin{verbatim}static PyObject *spam_system(PyObject *self, PyObject *args){    const char *command;    int sts;     if (!PyArg_ParseTuple(args, "s", &command))        return NULL;    sts = system(command);    return Py_BuildValue("i", sts);}\end{verbatim} There is a straightforward translation from the argument list inPython (for example, the single expression \code{"ls -l"}) to thearguments passed to the C function.  The C function always has twoarguments, conventionally named \var{self} and \var{args}. The \var{self} argument is only used when the C function implements abuilt-in method, not a function. In the example, \var{self} willalways be a \NULL{} pointer, since we are defining a function, not amethod.  (This is done so that the interpreter doesn't have tounderstand two different types of C functions.) The \var{args} argument will be a pointer to a Python tuple objectcontaining the arguments.  Each item of the tuple corresponds to anargument in the call's argument list.  The arguments are Pythonobjects --- in order to do anything with them in our C function we haveto convert them to C values.  The function \cfunction{PyArg_ParseTuple()}in the Python API checks the argument types and converts them to Cvalues.  It uses a template string to determine the required types ofthe arguments as well as the types of the C variables into which tostore the converted values.  More about this later. \cfunction{PyArg_ParseTuple()} returns true (nonzero) if all arguments havethe right type and its components have been stored in the variableswhose addresses are passed.  It returns false (zero) if an invalidargument list was passed.  In the latter case it also raises anappropriate exception so the calling function can return\NULL{} immediately (as we saw in the example).  \section{Intermezzo: Errors and Exceptions         \label{errors}} An important convention throughout the Python interpreter is thefollowing: when a function fails, it should set an exception conditionand return an error value (usually a \NULL{} pointer).  Exceptionsare stored in a static global variable inside the interpreter; if thisvariable is \NULL{} no exception has occurred.  A second globalvariable stores the ``associated value'' of the exception (the secondargument to \keyword{raise}).  A third variable contains the stacktraceback in case the error originated in Python code.  These threevariables are the C equivalents of the Python variables\code{sys.exc_type}, \code{sys.exc_value} and \code{sys.exc_traceback} (seethe section on module \module{sys} in the\citetitle[../lib/lib.html]{Python Library Reference}).  It isimportant to know about them to understand how errors are passedaround. The Python API defines a number of functions to set various types ofexceptions. The most common one is \cfunction{PyErr_SetString()}.  Its argumentsare an exception object and a C string.  The exception object isusually a predefined object like \cdata{PyExc_ZeroDivisionError}.  TheC string indicates the cause of the error and is converted to aPython string object and stored as the ``associated value'' of theexception. Another useful function is \cfunction{PyErr_SetFromErrno()}, which onlytakes an exception argument and constructs the associated value byinspection of the global variable \cdata{errno}.  The mostgeneral function is \cfunction{PyErr_SetObject()}, which takes two objectarguments, the exception and its associated value.  You don't need to\cfunction{Py_INCREF()} the objects passed to any of these functions. You can test non-destructively whether an exception has been set with\cfunction{PyErr_Occurred()}.  This returns the current exception object,or \NULL{} if no exception has occurred.  You normally don't needto call \cfunction{PyErr_Occurred()} to see whether an error occurred in afunction call, since you should be able to tell from the return value. When a function \var{f} that calls another function \var{g} detectsthat the latter fails, \var{f} should itself return an error value(usually \NULL{} or \code{-1}).  It should \emph{not} call one of the\cfunction{PyErr_*()} functions --- one has already been called by \var{g}.\var{f}'s caller is then supposed to also return an error indicationto \emph{its} caller, again \emph{without} calling \cfunction{PyErr_*()},and so on --- the most detailed cause of the error was alreadyreported by the function that first detected it.  Once the errorreaches the Python interpreter's main loop, this aborts the currentlyexecuting Python code and tries to find an exception handler specifiedby the Python programmer. (There are situations where a module can actually give a more detailederror message by calling another \cfunction{PyErr_*()} function, and insuch cases it is fine to do so.  As a general rule, however, this isnot necessary, and can cause information about the cause of the errorto be lost: most operations can fail for a variety of reasons.) To ignore an exception set by a function call that failed, the exceptioncondition must be cleared explicitly by calling \cfunction{PyErr_Clear()}. The only time C code should call \cfunction{PyErr_Clear()} is if it doesn'twant to pass the error on to the interpreter but wants to handle itcompletely by itself (possibly by trying something else, or pretendingnothing went wrong). Every failing \cfunction{malloc()} call must be turned into anexception --- the direct caller of \cfunction{malloc()} (or\cfunction{realloc()}) must call \cfunction{PyErr_NoMemory()} andreturn a failure indicator itself.  All the object-creating functions(for example, \cfunction{PyInt_FromLong()}) already do this, so thisnote is only relevant to those who call \cfunction{malloc()} directly. Also note that, with the important exception of\cfunction{PyArg_ParseTuple()} and friends, functions that return aninteger status usually return a positive value or zero for success and\code{-1} for failure, like \UNIX{} system calls. Finally, be careful to clean up garbage (by making\cfunction{Py_XDECREF()} or \cfunction{Py_DECREF()} calls for objectsyou have already created) when you return an error indicator! The choice of which exception to raise is entirely yours.  There arepredeclared C objects corresponding to all built-in Python exceptions,such as \cdata{PyExc_ZeroDivisionError}, which you can use directly.Of course, you should choose exceptions wisely --- don't use\cdata{PyExc_TypeError} to mean that a file couldn't be opened (thatshould probably be \cdata{PyExc_IOError}).  If something's wrong withthe argument list, the \cfunction{PyArg_ParseTuple()} function usuallyraises \cdata{PyExc_TypeError}.  If you have an argument whose valuemust be in a particular range or must satisfy other conditions,\cdata{PyExc_ValueError} is appropriate. You can also define a new exception that is unique to your module.For this, you usually declare a static object variable at thebeginning of your file: \begin{verbatim}static PyObject *SpamError;\end{verbatim} and initialize it in your module's initialization function(\cfunction{initspam()}) with an exception object (leaving outthe error checking for now): \begin{verbatim}PyMODINIT_FUNCinitspam(void){    PyObject *m;     m = Py_InitModule("spam", SpamMethods);    if (m == NULL)        return;     SpamError = PyErr_NewException("spam.error", NULL, NULL);    Py_INCREF(SpamError);    PyModule_AddObject(m, "error", SpamError);}\end{verbatim} Note that the Python name for the exception object is\exception{spam.error}.  The \cfunction{PyErr_NewException()} functionmay create a class with the base class being \exception{Exception}(unless another class is passed in instead of \NULL), described in the\citetitle[../lib/lib.html]{Python Library Reference} under ``Built-inExceptions.'' Note also that the \cdata{SpamError} variable retains a reference tothe newly created exception class; this is intentional!  Since theexception could be removed from the module by external code, an ownedreference to the class is needed to ensure that it will not bediscarded, causing \cdata{SpamError} to become a dangling pointer.Should it become a dangling pointer, C code which raises the exceptioncould cause a core dump or other unintended side effects. We discuss the use of PyMODINIT_FUNC as a function return type later in thissample. \section{Back to the Example         \label{backToExample}} Going back to our example function, you should now be able tounderstand this statement: \begin{verbatim}    if (!PyArg_ParseTuple(args, "s", &command))        return NULL;\end{verbatim} It returns \NULL{} (the error indicator for functions returningobject pointers) if an error is detected in the argument list, relyingon the exception set by \cfunction{PyArg_ParseTuple()}.  Otherwise thestring value of the argument has been copied to the local variable\cdata{command}.  This is a pointer assignment and you are not supposedto modify the string to which it points (so in Standard C, the variable\cdata{command} should properly be declared as \samp{const char*command}). The next statement is a call to the \UNIX{} function\cfunction{system()}, passing it the string we just got from\cfunction{PyArg_ParseTuple()}: \begin{verbatim}    sts = system(command);\end{verbatim} Our \function{spam.system()} function must return the value of\cdata{sts} as a Python object.  This is done using the function\cfunction{Py_BuildValue()}, which is something like the inverse of\cfunction{PyArg_ParseTuple()}: it takes a format string and anarbitrary number of C values, and returns a new Python object.More info on \cfunction{Py_BuildValue()} is given later. \begin{verbatim}    return Py_BuildValue("i", sts);\end{verbatim} In this case, it will return an integer object.  (Yes, even integersare objects on the heap in Python!) If you have a C function that returns no useful argument (a functionreturning \ctype{void}), the corresponding Python function must return\code{None}.   You need this idiom to do so (which is implemented by the\csimplemacro{Py_RETURN_NONE} macro): \begin{verbatim}    Py_INCREF(Py_None);    return Py_None;\end{verbatim} \cdata{Py_None} is the C name for the special Python object\code{None}.  It is a genuine Python object rather than a \NULL{}pointer, which means ``error'' in most contexts, as we have seen.  \section{The Module's Method Table and Initialization Function         \label{methodTable}} I promised to show how \cfunction{spam_system()} is called from Pythonprograms.  First, we need to list its name and address in a ``methodtable'': \begin{verbatim}static PyMethodDef SpamMethods[] = {    ...    {"system",  spam_system, METH_VARARGS,     "Execute a shell command."},    ...    {NULL, NULL, 0, NULL}        /* Sentinel */};\end{verbatim} Note the third entry (\samp{METH_VARARGS}).  This is a flag tellingthe interpreter the calling convention to be used for the Cfunction.  It should normally always be \samp{METH_VARARGS} or\samp{METH_VARARGS | METH_KEYWORDS}; a value of \code{0} means that anobsolete variant of \cfunction{PyArg_ParseTuple()} is used. When using only \samp{METH_VARARGS}, the function should expectthe Python-level parameters to be passed in as a tuple acceptable forparsing via \cfunction{PyArg_ParseTuple()}; more information on thisfunction is provided below. The \constant{METH_KEYWORDS} bit may be set in the third field ifkeyword arguments should be passed to the function.  In this case, theC function should accept a third \samp{PyObject *} parameter whichwill be a dictionary of keywords.  Use\cfunction{PyArg_ParseTupleAndKeywords()} to parse the arguments tosuch a function. The method table must be passed to the interpreter in the module'sinitialization function.  The initialization function must be named\cfunction{init\var{name}()}, where \var{name} is the name of themodule, and should be the only non-\keyword{static} item defined inthe module file: \begin{verbatim}PyMODINIT_FUNCinitspam(void){    (void) Py_InitModule("spam", SpamMethods);}\end{verbatim} Note that PyMODINIT_FUNC declares the function as \code{void} return type, declares any special linkage declarations required by the platform, and for \Cpp{} declares the function as \code{extern "C"}. When the Python program imports module \module{spam} for the firsttime, \cfunction{initspam()} is called. (See below for comments aboutembedding Python.)  It calls\cfunction{Py_InitModule()}, which creates a ``module object'' (whichis inserted in the dictionary \code{sys.modules} under the key\code{"spam"}), and inserts built-in function objects into the newlycreated module based upon the table (an array of \ctype{PyMethodDef}structures) that was passed as its second argument.\cfunction{Py_InitModule()} returns a pointer to the module objectthat it creates (which is unused here).  It may abort with a fatal errorfor certain errors, or return \NULL{} if the module could not beinitialized satisfactorily. When embedding Python, the \cfunction{initspam()} function is notcalled automatically unless there's an entry in the\cdata{_PyImport_Inittab} table.  The easiest way to handle this is to statically initialize your statically-linked modules by directlycalling \cfunction{initspam()} after the call to\cfunction{Py_Initialize()}: \begin{verbatim}intmain(int argc, char *argv[]){    /* Pass argv[0] to the Python interpreter */    Py_SetProgramName(argv[0]);     /* Initialize the Python interpreter.  Required. */    Py_Initialize();     /* Add a static module */    initspam();\end{verbatim} An example may be found in the file \file{Demo/embed/demo.c} in thePython source distribution. \note{Removing entries from \code{sys.modules} or importingcompiled modules into multiple interpreters within a process (orfollowing a \cfunction{fork()} without an intervening\cfunction{exec()}) can create problems for some extension modules.Extension module authors should exercise caution when initializinginternal data structures.Note also that the \function{reload()} function can be used withextension modules, and will call the module initialization function(\cfunction{initspam()} in the example), but will not load the moduleagain if it was loaded from a dynamically loadable object file(\file{.so} on \UNIX, \file{.dll} on Windows).} A more substantial example module is included in the Python sourcedistribution as \file{Modules/xxmodule.c}.  This file may be used as a template or simply read as an example.  The \program{modulator.py}script included in the source distribution or Windows install provides a simple graphical user interface for declaring the functions andobjects which a module should implement, and can generate a templatewhich can be filled in.  The script lives in the\file{Tools/modulator/} directory; see the \file{README} file therefor more information.  \section{Compilation and Linkage         \label{compilation}} There are two more things to do before you can use your new extension:compiling and linking it with the Python system.  If you use dynamicloading, the details may depend on the style of dynamic loading yoursystem uses; see the chapters about building extension modules(chapter \ref{building}) and additional information that pertains onlyto building on Windows (chapter \ref{building-on-windows}) for moreinformation about this. If you can't use dynamic loading, or if you want to make your module apermanent part of the Python interpreter, you will have to change theconfiguration setup and rebuild the interpreter.  Luckily, this isvery simple on \UNIX: just place your file (\file{spammodule.c} forexample) in the \file{Modules/} directory of an unpacked sourcedistribution, add a line to the file \file{Modules/Setup.local}describing your file: \begin{verbatim}spam spammodule.o\end{verbatim} and rebuild the interpreter by running \program{make} in the topleveldirectory.  You can also run \program{make} in the \file{Modules/}subdirectory, but then you must first rebuild \file{Makefile}there by running `\program{make} Makefile'.  (This is necessary eachtime you change the \file{Setup} file.) If your module requires additional libraries to link with, these canbe listed on the line in the configuration file as well, for instance: \begin{verbatim}spam spammodule.o -lX11\end{verbatim} \section{Calling Python Functions from C         \label{callingPython}} So far we have concentrated on making C functions callable fromPython.  The reverse is also useful: calling Python functions from C.This is especially the case for libraries that support so-called``callback'' functions.  If a C interface makes use of callbacks, theequivalent Python often needs to provide a callback mechanism to thePython programmer; the implementation will require calling the Pythoncallback functions from a C callback.  Other uses are also imaginable. Fortunately, the Python interpreter is easily called recursively, andthere is a standard interface to call a Python function.  (I won'tdwell on how to call the Python parser with a particular string asinput --- if you're interested, have a look at the implementation ofthe \programopt{-c} command line option in \file{Python/pythonmain.c}from the Python source code.) Calling a Python function is easy.  First, the Python program mustsomehow pass you the Python function object.  You should provide afunction (or some other interface) to do this.  When this function iscalled, save a pointer to the Python function object (be careful to\cfunction{Py_INCREF()} it!) in a global variable --- or wherever yousee fit. For example, the following function might be part of a moduledefinition: \begin{verbatim}static PyObject *my_callback = NULL; static PyObject *my_set_callback(PyObject *dummy, PyObject *args){    PyObject *result = NULL;    PyObject *temp;     if (PyArg_ParseTuple(args, "O:set_callback", &temp)) {        if (!PyCallable_Check(temp)) {            PyErr_SetString(PyExc_TypeError, "parameter must be callable");            return NULL;        }        Py_XINCREF(temp);         /* Add a reference to new callback */        Py_XDECREF(my_callback);  /* Dispose of previous callback */        my_callback = temp;       /* Remember new callback */        /* Boilerplate to return "None" */

Lines 1–500 of 1390. Load more