<HTML> <HEAD><TITLE>Metaclasses in Python 1.5</TITLE></HEAD> <BODY BGCOLOR="FFFFFF"> <H1>Metaclasses in Python 1.5</H1><H2>(A.k.a. The Killer Joke :-)</H2> <HR> (<i>Postscript:</i> reading this essay is probably not the best way tounderstand the metaclass hook described here.  See a <AHREF="meta-vladimir.txt">message posted by Vladimir Marangozov</A>which may give a gentler introduction to the matter.  You may alsowant to search Deja News for messages with "metaclass" in the subjectposted to comp.lang.python in July and August 1998.) <HR> <P>In previous Python releases (and still in 1.5), there is somethingcalled the ``Don Beaudry hook'', after its inventor and champion.This allows C extensions to provide alternate class behavior, therebyallowing the Python class syntax to be used to define other class-likeentities.  Don Beaudry has used this in his infamous <AHREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> package; JimFulton has used it in his <AHREF="http://www.digicool.com/releases/ExtensionClass/">ExtensionClasses</A> package.  (It has also been referred to as the ``DonBeaudry <i>hack</i>,'' but that's a misnomer.  There's nothing hackishabout it -- in fact, it is rather elegant and deep, even thoughthere's something dark to it.) <P>(On first reading, you may want to skip directly to the examples inthe section "Writing Metaclasses in Python" below, unless you wantyour head to explode.) <P> <HR> <P>Documentation of the Don Beaudry hook has purposefully been keptminimal, since it is a feature of incredible power, and is easilyabused.  Basically, it checks whether the <b>type of the baseclass</b> is callable, and if so, it is called to create the newclass. <P>Note the two indirection levels.  Take a simple example: <PRE>class B:    pass class C(B):    pass</PRE> Take a look at the second class definition, and try to fathom ``thetype of the base class is callable.'' <P>(Types are not classes, by the way.  See questions 4.2, 4.19 and inparticular 6.22 in the <AHREF="http://www.python.org/cgi-bin/faqw.py" >Python FAQ</A>for more on this topic.) <P> <UL> <LI>The <b>base class</b> is B; this one's easy.<P> <LI>Since B is a class, its type is ``class''; so the <b>type of thebase class</b> is the type ``class''.  This is also known astypes.ClassType, assuming the standard module <code>types</code> hasbeen imported.<P> <LI>Now is the type ``class'' <b>callable</b>?  No, because types (incore Python) are never callable.  Classes are callable (calling aclass creates a new instance) but types aren't.<P> </UL> <P>So our conclusion is that in our example, the type of the baseclass (of C) is not callable.  So the Don Beaudry hook does not apply,and the default class creation mechanism is used (which is also usedwhen there is no base class).  In fact, the Don Beaudry hook neverapplies when using only core Python, since the type of a core objectis never callable. <P>So what do Don and Jim do in order to use Don's hook?  Write anextension that defines at least two new Python object types.  Thefirst would be the type for ``class-like'' objects usable as a baseclass, to trigger Don's hook.  This type must be made callable.That's why we need a second type.  Whether an object is callabledepends on its type.  So whether a type object is callable depends on<i>its</i> type, which is a <i>meta-type</i>.  (In core Python thereis only one meta-type, the type ``type'' (types.TypeType), which isthe type of all type objects, even itself.)  A new meta-type mustbe defined that makes the type of the class-like objects callable.(Normally, a third type would also be needed, the new ``instance''type, but this is not an absolute requirement -- the new class typecould return an object of some existing type when invoked to create aninstance.) <P>Still confused?  Here's a simple device due to Don himself toexplain metaclasses.  Take a simple class definition; assume B is aspecial class that triggers Don's hook: <PRE>class C(B):    a = 1    b = 2</PRE> This can be though of as equivalent to: <PRE>C = type(B)('C', (B,), {'a': 1, 'b': 2})</PRE> If that's too dense for you, here's the same thing written out usingtemporary variables: <PRE>creator = type(B)               # The type of the base classname = 'C'                      # The name of the new classbases = (B,)                    # A tuple containing the base class(es)namespace = {'a': 1, 'b': 2}    # The namespace of the class statementC = creator(name, bases, namespace)</PRE> This is analogous to what happens without the Don Beaudry hook, exceptthat in that case the creator function is set to the default classcreator. <P>In either case, the creator is called with three arguments.  Thefirst one, <i>name</i>, is the name of the new class (as given at thetop of the class statement).  The <i>bases</i> argument is a tuple ofbase classes (a singleton tuple if there's only one base class, likethe example).  Finally, <i>namespace</i> is a dictionary containingthe local variables collected during execution of the class statement. <P>Note that the contents of the namespace dictionary is simplywhatever names were defined in the class statement.  A little-knownfact is that when Python executes a class statement, it enters a newlocal namespace, and all assignments and function definitions takeplace in this namespace.  Thus, after executing the following classstatement: <PRE>class C:    a = 1    def f(s): pass</PRE> the class namespace's contents would be {'a': 1, 'f': &lt;function f...&gt;}. <P>But enough already about writing Python metaclasses in C; read thedocumentation of <AHREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> or <AHREF="http://www.digicool.com/papers/ExtensionClass.html" >ExtensionClasses</A> for more information. <P> <HR> <H2>Writing Metaclasses in Python</H2> <P>In Python 1.5, the requirement to write a C extension in order towrite metaclasses has been dropped (though you can still doit, of course).  In addition to the check ``is the type of the baseclass callable,'' there's a check ``does the base class have a__class__ attribute.''  If so, it is assumed that the __class__attribute refers to a class. <P>Let's repeat our simple example from above: <PRE>class C(B):    a = 1    b = 2</PRE> Assuming B has a __class__ attribute, this translates into: <PRE>C = B.__class__('C', (B,), {'a': 1, 'b': 2})</PRE> This is exactly the same as before except that instead of type(B),B.__class__ is invoked.  If you have read <A HREF="http://www.python.org/cgi-bin/faqw.py?req=show&file=faq06.022.htp">FAQ question 6.22</A> you will understand that while there is a bigtechnical difference between type(B) and B.__class__, they play thesame role at different abstraction levels.  And perhaps at some pointin the future they will really be the same thing (at which point youwould be able to derive subclasses from built-in types). <P>At this point it may be worth mentioning that C.__class__ is thesame object as B.__class__, i.e., C's metaclass is the same as B'smetaclass.  In other words, subclassing an existing class creates anew (meta)inststance of the base class's metaclass. <P>Going back to the example, the class B.__class__ is instantiated,passing its constructor the same three arguments that are passed tothe default class constructor or to an extension's metaclass:<i>name</i>, <i>bases</i>, and <i>namespace</i>. <P>It is easy to be confused by what exactly happens when using ametaclass, because we lose the absolute distinction between classesand instances: a class is an instance of a metaclass (a``metainstance''), but technically (i.e. in the eyes of the pythonruntime system), the metaclass is just a class, and the metainstanceis just an instance.  At the end of the class statement, the metaclasswhose metainstance is used as a base class is instantiated, yielding asecond metainstance (of the same metaclass).  This metainstance isthen used as a (normal, non-meta) class; instantiation of the classmeans calling the metainstance, and this will return a real instance.And what class is that an instance of?  Conceptually, it is of coursean instance of our metainstance; but in most cases the Python runtimesystem will see it as an instance of a a helper class used by themetaclass to implement its (non-meta) instances... <P>Hopefully an example will make things clearer.  Let's presume wehave a metaclass MetaClass1.  It's helper class (for non-metainstances) is callled HelperClass1.  We now (manually) instantiateMetaClass1 once to get an empty special base class: <PRE>BaseClass1 = MetaClass1("BaseClass1", (), {})</PRE> We can now use BaseClass1 as a base class in a class statement: <PRE>class MySpecialClass(BaseClass1):    i = 1    def f(s): pass</PRE> At this point, MySpecialClass is defined; it is a metainstance ofMetaClass1 just like BaseClass1, and in fact the expression``BaseClass1.__class__ == MySpecialClass.__class__ == MetaClass1''yields true. <P>We are now ready to create instances of MySpecialClass.  Let'sassume that no constructor arguments are required: <PRE>x = MySpecialClass()y = MySpecialClass()print x.__class__, y.__class__</PRE> The print statement shows that x and y are instances of HelperClass1.How did this happen?  MySpecialClass is an instance of MetaClass1(``meta'' is irrelevant here); when an instance is called, its__call__ method is invoked, and presumably the __call__ method definedby MetaClass1 returns an instance of HelperClass1. <P>Now let's see how we could use metaclasses -- what can we dowith metaclasses that we can't easily do without them?  Here's oneidea: a metaclass could automatically insert trace calls for allmethod calls.  Let's first develop a simplified example, withoutsupport for inheritance or other ``advanced'' Python features (we'lladd those later). <PRE>import types class Tracing:    def __init__(self, name, bases, namespace):        """Create a new class."""        self.__name__ = name        self.__bases__ = bases        self.__namespace__ = namespace    def __call__(self):        """Create a new instance."""        return Instance(self) class Instance:    def __init__(self, klass):        self.__klass__ = klass    def __getattr__(self, name):        try:            value = self.__klass__.__namespace__[name]        except KeyError:            raise AttributeError, name        if type(value) is not types.FunctionType:            return value        return BoundMethod(value, self) class BoundMethod:    def __init__(self, function, instance):        self.function = function        self.instance = instance    def __call__(self, *args):        print "calling", self.function, "for", self.instance, "with", args        return apply(self.function, (self.instance,) + args) Trace = Tracing('Trace', (), {}) class MyTracedClass(Trace):    def method1(self, a):        self.a = a    def method2(self):        return self.a aninstance = MyTracedClass() aninstance.method1(10) print "the answer is %d" % aninstance.method2()</PRE> Confused already?  The intention is to read this from top down.  TheTracing class is the metaclass we're defining.  Its structure isreally simple. <P> <UL> <LI>The __init__ method is invoked when a new Tracing instance iscreated, e.g. the definition of class MyTracedClass later in theexample.  It simply saves the class name, base classes and namespaceas instance variables.<P> <LI>The __call__ method is invoked when a Tracing instance is called,e.g. the creation of aninstance later in the example.  It returns aninstance of the class Instance, which is defined next.<P> </UL> <P>The class Instance is the class used for all instances of classesbuilt using the Tracing metaclass, e.g. aninstance.  It has twomethods: <P> <UL> <LI>The __init__ method is invoked from the Tracing.__call__ methodabove to initialize a new instance.  It saves the class reference asan instance variable.  It uses a funny name because the user'sinstance variables (e.g. self.a later in the example) live in the samenamespace.<P> <LI>The __getattr__ method is invoked whenever the user codereferences an attribute of the instance that is not an instancevariable (nor a class variable; but except for __init__ and__getattr__ there are no class variables).  It will be called, forexample, when aninstance.method1 is referenced in the example, withself set to aninstance and name set to the string "method1".<P> </UL> <P>The __getattr__ method looks the name up in the __namespace__dictionary.  If it isn't found, it raises an AttributeError exception.(In a more realistic example, it would first have to look through thebase classes as well.)  If it is found, there are two possibilities:it's either a function or it isn't.  If it's not a function, it isassumed to be a class variable, and its value is returned.  If it's afunction, we have to ``wrap'' it in instance of yet another helperclass, BoundMethod. <P>The BoundMethod class is needed to implement a familiar feature:when a method is defined, it has an initial argument, self, which isautomatically bound to the relevant instance when it is called.  Forexample, aninstance.method1(10) is equivalent to method1(aninstance,10).  In the example if this call, first a temporary BoundMethodinstance is created with the following constructor call: temp =BoundMethod(method1, aninstance); then this instance is called astemp(10).  After the call, the temporary instance is discarded. <P> <UL> <LI>The __init__ method is invoked for the constructor callBoundMethod(method1, aninstance).  It simply saves away itsarguments.<P> <LI>The __call__ method is invoked when the bound method instance iscalled, as in temp(10).  It needs to call method1(aninstance, 10).However, even though self.function is now method1 and self.instance isaninstance, it can't call self.function(self.instance, args) directly,because it should work regardless of the number of arguments passed.(For simplicity, support for keyword arguments has been omitted.)<P> </UL> <P>In order to be able to support arbitrary argument lists, the__call__ method first constructs a new argument tuple.  Conveniently,because of the notation *args in __call__'s own argument list, thearguments to __call__ (except for self) are placed in the tuple args.To construct the desired argument list, we concatenate a singletontuple containing the instance with the args tuple: (self.instance,) +args.  (Note the trailing comma used to construct the singletontuple.)  In our example, the resulting argument tuple is (aninstance,10). <P>The intrinsic function apply() takes a function and an argumenttuple and calls the function for it.  In our example, we are callingapply(method1, (aninstance, 10)) which is equivalent to callingmethod(aninstance, 10). <P>From here on, things should come together quite easily.  The outputof the example code is something like this: <PRE>calling &lt;function method1 at ae8d8&gt; for &lt;Instance instance at 95ab0&gt; with (10,)calling &lt;function method2 at ae900&gt; for &lt;Instance instance at 95ab0&gt; with ()the answer is 10</PRE> <P>That was about the shortest meaningful example that I could come upwith.  A real tracing metaclass (for example, <AHREF="#Trace">Trace.py</A> discussed below) needs to be morecomplicated in two dimensions. <P>First, it needs to support more advanced Python features such asclass variables, inheritance, __init__ methods, and keyword arguments. <P>Second, it needs to provide a more flexible way to handle theactual tracing information; perhaps it should be possible to writeyour own tracing function that gets called, perhaps it should bepossible to enable and disable tracing on a per-class or per-instancebasis, and perhaps a filter so that only interesting calls are traced;it should also be able to trace the return value of the call (or theexception it raised if an error occurs).  Even the Trace.py exampledoesn't support all these features yet. <P> <HR> <H1>Real-life Examples</H1> <P>Have a look at some very preliminary examples that I coded up toteach myself how to write metaclasses: <DL> <DT><A HREF="Enum.py">Enum.py</A> <DD>This (ab)uses the class syntax as an elegant way to defineenumerated types.  The resulting classes are never instantiated --rather, their class attributes are the enumerated values.  Forexample: <PRE>class Color(Enum):    red = 1    green = 2    blue = 3print Color.red</PRE> will print the string ``Color.red'', while ``Color.red==1'' is true,and ``Color.red + 1'' raise a TypeError exception. <P> <DT><A NAME=Trace></A><A HREF="Trace.py">Trace.py</A> <DD>The resulting classes work much like standardclasses, but by setting a special class or instance attribute__trace_output__ to point to a file, all calls to the class's methodsare traced.  It was a bit of a struggle to get this right.  Thisshould probably redone using the generic metaclass below. <P> <DT><A HREF="Meta.py">Meta.py</A> <DD>A generic metaclass.  This is an attempt at finding out how muchstandard class behavior can be mimicked by a metaclass.  Thepreliminary answer appears to be that everything's fine as long as theclass (or its clients) don't look at the instance's __class__attribute, nor at the class's __dict__ attribute.  The use of__getattr__ internally makes the classic implementation of __getattr__hooks tough; we provide a similar hook _getattr_ instead.(__setattr__ and __delattr__ are not affected.)(XXX Hm.  Could detect presence of __getattr__ and rename it.) <P> <DT><A HREF="Eiffel.py">Eiffel.py</A> <DD>Uses the above generic metaclass to implement Eiffel stylepre-conditions and post-conditions. <P> <DT><A HREF="Synch.py">Synch.py</A>

Lines 1–500 of 605. Load more