'''"Executable documentation" for the pickle module. Extensive comments about the pickle protocols and pickle-machine opcodescan be found here.  Some functions meant for external use: genops(pickle)   Generate all the opcodes in a pickle, as (opcode, arg, position) triples. dis(pickle, out=None, memo=None, indentlevel=4)   Print a symbolic disassembly of a pickle.''' __all__ = ['dis',           'genops',          ] # Other ideas:## - A pickle verifier:  read a pickle and check it exhaustively for#   well-formedness.  dis() does a lot of this already.## - A protocol identifier:  examine a pickle and return its protocol number#   (== the highest .proto attr value among all the opcodes in the pickle).#   dis() already prints this info at the end.## - A pickle optimizer:  for example, tuple-building code is sometimes more#   elaborate than necessary, catering for the possibility that the tuple#   is recursive.  Or lots of times a PUT is generated that's never accessed#   by a later GET.  """"A pickle" is a program for a virtual pickle machine (PM, but more accuratelycalled an unpickling machine).  It's a sequence of opcodes, interpreted by thePM, building an arbitrarily complex Python object. For the most part, the PM is very simple:  there are no looping, testing, orconditional instructions, no arithmetic and no function calls.  Opcodes areexecuted once each, from first to last, until a STOP opcode is reached. The PM has two data areas, "the stack" and "the memo". Many opcodes push Python objects onto the stack; e.g., INT pushes a Pythoninteger object on the stack, whose value is gotten from a decimal stringliteral immediately following the INT opcode in the pickle bytestream.  Otheropcodes take Python objects off the stack.  The result of unpickling iswhatever object is left on the stack when the final STOP opcode is executed. The memo is simply an array of objects, or it can be implemented as a dictmapping little integers to objects.  The memo serves as the PM's "long termmemory", and the little integers indexing the memo are akin to variablenames.  Some opcodes pop a stack object into the memo at a given index,and others push a memo object at a given index onto the stack again. At heart, that's all the PM has.  Subtleties arise for these reasons: + Object identity.  Objects can be arbitrarily complex, and subobjects  may be shared (for example, the list [a, a] refers to the same object a  twice).  It can be vital that unpickling recreate an isomorphic object  graph, faithfully reproducing sharing. + Recursive objects.  For example, after "L = []; L.append(L)", L is a  list, and L[0] is the same list.  This is related to the object identity  point, and some sequences of pickle opcodes are subtle in order to  get the right result in all cases. + Things pickle doesn't know everything about.  Examples of things pickle  does know everything about are Python's builtin scalar and container  types, like ints and tuples.  They generally have opcodes dedicated to  them.  For things like module references and instances of user-defined  classes, pickle's knowledge is limited.  Historically, many enhancements  have been made to the pickle protocol in order to do a better (faster,  and/or more compact) job on those. + Backward compatibility and micro-optimization.  As explained below,  pickle opcodes never go away, not even when better ways to do a thing  get invented.  The repertoire of the PM just keeps growing over time.  For example, protocol 0 had two opcodes for building Python integers (INT  and LONG), protocol 1 added three more for more-efficient pickling of short  integers, and protocol 2 added two more for more-efficient pickling of  long integers (before protocol 2, the only ways to pickle a Python long  took time quadratic in the number of digits, for both pickling and  unpickling).  "Opcode bloat" isn't so much a subtlety as a source of  wearying complication.  Pickle protocols: For compatibility, the meaning of a pickle opcode never changes.  Instead newpickle opcodes get added, and each version's unpickler can handle all thepickle opcodes in all protocol versions to date.  So old pickles continue tobe readable forever.  The pickler can generally be told to restrict itself tothe subset of opcodes available under previous protocol versions too, so thatusers can create pickles under the current version readable by olderversions.  However, a pickle does not contain its version number embeddedwithin it.  If an older unpickler tries to read a pickle using a laterprotocol, the result is most likely an exception due to seeing an unknown (inthe older unpickler) opcode. The original pickle used what's now called "protocol 0", and what was called"text mode" before Python 2.3.  The entire pickle bytestream is made up ofprintable 7-bit ASCII characters, plus the newline character, in protocol 0.That's why it was called text mode.  Protocol 0 is small and elegant, butsometimes painfully inefficient. The second major set of additions is now called "protocol 1", and was called"binary mode" before Python 2.3.  This added many opcodes with argumentsconsisting of arbitrary bytes, including NUL bytes and unprintable "high bit"bytes.  Binary mode pickles can be substantially smaller than equivalenttext mode pickles, and sometimes faster too; e.g., BININT represents a 4-byteint as 4 bytes following the opcode, which is cheaper to unpickle than the(perhaps) 11-character decimal string attached to INT.  Protocol 1 also addeda number of opcodes that operate on many stack elements at once (like APPENDSand SETITEMS), and "shortcut" opcodes (like EMPTY_DICT and EMPTY_TUPLE). The third major set of additions came in Python 2.3, and is called "protocol2".  This added: - A better way to pickle instances of new-style classes (NEWOBJ). - A way for a pickle to identify its protocol (PROTO). - Time- and space- efficient pickling of long ints (LONG{1,4}). - Shortcuts for small tuples (TUPLE{1,2,3}}. - Dedicated opcodes for bools (NEWTRUE, NEWFALSE). - The "extension registry", a vector of popular objects that can be pushed  efficiently by index (EXT{1,2,4}).  This is akin to the memo and GET, but  the registry contents are predefined (there's nothing akin to the memo's  PUT). Another independent change with Python 2.3 is the abandonment of anypretense that it might be safe to load pickles received from untrustedparties -- no sufficient security analysis has been done to guaranteethis and there isn't a use case that warrants the expense of such ananalysis. To this end, all tests for __safe_for_unpickling__ or forcopy_reg.safe_constructors are removed from the unpickling code.References to these variables in the descriptions below are to be seenas describing unpickling in Python 2.2 and before.""" # Meta-rule:  Descriptions are stored in instances of descriptor objects,# with plain constructors.  No meta-language is defined from which# descriptors could be constructed.  If you want, e.g., XML, write a little# program to generate XML from the objects. ############################################################################### Some pickle opcodes have an argument, following the opcode in the# bytestream.  An argument is of a specific type, described by an instance# of ArgumentDescriptor.  These are not to be confused with arguments taken# off the stack -- ArgumentDescriptor applies only to arguments embedded in# the opcode stream, immediately following an opcode. # Represents the number of bytes consumed by an argument delimited by the# next newline character.UP_TO_NEWLINE = -1 # Represents the number of bytes consumed by a two-argument opcode where# the first argument gives the number of bytes in the second argument.TAKEN_FROM_ARGUMENT1 = -2   # num bytes is 1-byte unsigned intTAKEN_FROM_ARGUMENT4 = -3   # num bytes is 4-byte signed little-endian int class ArgumentDescriptor(object):    __slots__ = (        # name of descriptor record, also a module global name; a string        'name',         # length of argument, in bytes; an int; UP_TO_NEWLINE and        # TAKEN_FROM_ARGUMENT{1,4} are negative values for variable-length        # cases        'n',         # a function taking a file-like object, reading this kind of argument        # from the object at the current position, advancing the current        # position by n bytes, and returning the value of the argument        'reader',         # human-readable docs for this arg descriptor; a string        'doc',    )     def __init__(self, name, n, reader, doc):        assert isinstance(name, str)        self.name = name         assert isinstance(n, int) and (n >= 0 or                                       n in (UP_TO_NEWLINE,                                             TAKEN_FROM_ARGUMENT1,                                             TAKEN_FROM_ARGUMENT4))        self.n = n         self.reader = reader         assert isinstance(doc, str)        self.doc = doc from struct import unpack as _unpack def read_uint1(f):    r"""    >>> import StringIO    >>> read_uint1(StringIO.StringIO('\xff'))    255    """     data = f.read(1)    if data:        return ord(data)    raise ValueError("not enough data in stream to read uint1") uint1 = ArgumentDescriptor(            name='uint1',            n=1,            reader=read_uint1,            doc="One-byte unsigned integer.")  def read_uint2(f):    r"""    >>> import StringIO    >>> read_uint2(StringIO.StringIO('\xff\x00'))    255    >>> read_uint2(StringIO.StringIO('\xff\xff'))    65535    """     data = f.read(2)    if len(data) == 2:        return _unpack("<H", data)[0]    raise ValueError("not enough data in stream to read uint2") uint2 = ArgumentDescriptor(            name='uint2',            n=2,            reader=read_uint2,            doc="Two-byte unsigned integer, little-endian.")  def read_int4(f):    r"""    >>> import StringIO    >>> read_int4(StringIO.StringIO('\xff\x00\x00\x00'))    255    >>> read_int4(StringIO.StringIO('\x00\x00\x00\x80')) == -(2**31)    True    """     data = f.read(4)    if len(data) == 4:        return _unpack("<i", data)[0]    raise ValueError("not enough data in stream to read int4") int4 = ArgumentDescriptor(           name='int4',           n=4,           reader=read_int4,           doc="Four-byte signed integer, little-endian, 2's complement.")  def read_stringnl(f, decode=True, stripquotes=True):    r"""    >>> import StringIO    >>> read_stringnl(StringIO.StringIO("'abcd'\nefg\n"))    'abcd'     >>> read_stringnl(StringIO.StringIO("\n"))    Traceback (most recent call last):    ...    ValueError: no string quotes around ''     >>> read_stringnl(StringIO.StringIO("\n"), stripquotes=False)    ''     >>> read_stringnl(StringIO.StringIO("''\n"))    ''     >>> read_stringnl(StringIO.StringIO('"abcd"'))    Traceback (most recent call last):    ...    ValueError: no newline found when trying to read stringnl     Embedded escapes are undone in the result.    >>> read_stringnl(StringIO.StringIO(r"'a\n\\b\x00c\td'" + "\n'e'"))    'a\n\\b\x00c\td'    """     data = f.readline()    if not data.endswith('\n'):        raise ValueError("no newline found when trying to read stringnl")    data = data[:-1]    # lose the newline     if stripquotes:        for q in "'\"":            if data.startswith(q):                if not data.endswith(q):                    raise ValueError("strinq quote %r not found at both "                                     "ends of %r" % (q, data))                data = data[1:-1]                break        else:            raise ValueError("no string quotes around %r" % data)     # I'm not sure when 'string_escape' was added to the std codecs; it's    # crazy not to use it if it's there.    if decode:        data = data.decode('string_escape')    return data stringnl = ArgumentDescriptor(               name='stringnl',               n=UP_TO_NEWLINE,               reader=read_stringnl,               doc="""A newline-terminated string.                    This is a repr-style string, with embedded escapes, and                   bracketing quotes.                   """) def read_stringnl_noescape(f):    return read_stringnl(f, decode=False, stripquotes=False) stringnl_noescape = ArgumentDescriptor(                        name='stringnl_noescape',                        n=UP_TO_NEWLINE,                        reader=read_stringnl_noescape,                        doc="""A newline-terminated string.                         This is a str-style string, without embedded escapes,                        or bracketing quotes.  It should consist solely of                        printable ASCII characters.                        """) def read_stringnl_noescape_pair(f):    r"""    >>> import StringIO    >>> read_stringnl_noescape_pair(StringIO.StringIO("Queue\nEmpty\njunk"))    'Queue Empty'    """     return "%s %s" % (read_stringnl_noescape(f), read_stringnl_noescape(f)) stringnl_noescape_pair = ArgumentDescriptor(                             name='stringnl_noescape_pair',                             n=UP_TO_NEWLINE,                             reader=read_stringnl_noescape_pair,                             doc="""A pair of newline-terminated strings.                              These are str-style strings, without embedded                             escapes, or bracketing quotes.  They should                             consist solely of printable ASCII characters.                             The pair is returned as a single string, with                             a single blank separating the two strings.                             """) def read_string4(f):    r"""    >>> import StringIO    >>> read_string4(StringIO.StringIO("\x00\x00\x00\x00abc"))    ''    >>> read_string4(StringIO.StringIO("\x03\x00\x00\x00abcdef"))    'abc'    >>> read_string4(StringIO.StringIO("\x00\x00\x00\x03abcdef"))    Traceback (most recent call last):    ...    ValueError: expected 50331648 bytes in a string4, but only 6 remain    """     n = read_int4(f)    if n < 0:        raise ValueError("string4 byte count < 0: %d" % n)    data = f.read(n)    if len(data) == n:        return data    raise ValueError("expected %d bytes in a string4, but only %d remain" %                     (n, len(data))) string4 = ArgumentDescriptor(              name="string4",              n=TAKEN_FROM_ARGUMENT4,              reader=read_string4,              doc="""A counted string.               The first argument is a 4-byte little-endian signed int giving              the number of bytes in the string, and the second argument is              that many bytes.              """)  def read_string1(f):    r"""    >>> import StringIO    >>> read_string1(StringIO.StringIO("\x00"))    ''    >>> read_string1(StringIO.StringIO("\x03abcdef"))    'abc'    """     n = read_uint1(f)    assert n >= 0    data = f.read(n)    if len(data) == n:        return data    raise ValueError("expected %d bytes in a string1, but only %d remain" %                     (n, len(data))) string1 = ArgumentDescriptor(              name="string1",              n=TAKEN_FROM_ARGUMENT1,              reader=read_string1,              doc="""A counted string.               The first argument is a 1-byte unsigned int giving the number              of bytes in the string, and the second argument is that many              bytes.              """)  def read_unicodestringnl(f):    r"""    >>> import StringIO    >>> read_unicodestringnl(StringIO.StringIO("abc\uabcd\njunk"))    u'abc\uabcd'    """     data = f.readline()    if not data.endswith('\n'):        raise ValueError("no newline found when trying to read "                         "unicodestringnl")    data = data[:-1]    # lose the newline    return unicode(data, 'raw-unicode-escape') unicodestringnl = ArgumentDescriptor(                      name='unicodestringnl',                      n=UP_TO_NEWLINE,                      reader=read_unicodestringnl,                      doc="""A newline-terminated Unicode string.                       This is raw-unicode-escape encoded, so consists of                      printable ASCII characters, and may contain embedded                      escape sequences.                      """) def read_unicodestring4(f):    r"""    >>> import StringIO    >>> s = u'abcd\uabcd'    >>> enc = s.encode('utf-8')    >>> enc    'abcd\xea\xaf\x8d'    >>> n = chr(len(enc)) + chr(0) * 3  # little-endian 4-byte length    >>> t = read_unicodestring4(StringIO.StringIO(n + enc + 'junk'))    >>> s == t    True     >>> read_unicodestring4(StringIO.StringIO(n + enc[:-1]))    Traceback (most recent call last):    ...    ValueError: expected 7 bytes in a unicodestring4, but only 6 remain    """     n = read_int4(f)    if n < 0:        raise ValueError("unicodestring4 byte count < 0: %d" % n)    data = f.read(n)    if len(data) == n:        return unicode(data, 'utf-8')    raise ValueError("expected %d bytes in a unicodestring4, but only %d "                     "remain" % (n, len(data))) unicodestring4 = ArgumentDescriptor(                    name="unicodestring4",                    n=TAKEN_FROM_ARGUMENT4,                    reader=read_unicodestring4,                    doc="""A counted Unicode string.                     The first argument is a 4-byte little-endian signed int                    giving the number of bytes in the string, and the second                    argument-- the UTF-8 encoding of the Unicode string --                    contains that many bytes.                    """)  def read_decimalnl_short(f):    r"""    >>> import StringIO    >>> read_decimalnl_short(StringIO.StringIO("1234\n56"))    1234     >>> read_decimalnl_short(StringIO.StringIO("1234L\n56"))    Traceback (most recent call last):    ...    ValueError: trailing 'L' not allowed in '1234L'    """     s = read_stringnl(f, decode=False, stripquotes=False)    if s.endswith("L"):

Lines 1–500 of 2246. Load more