\section{\module{codecs} ---         Codec registry and base classes} \declaremodule{standard}{codecs}\modulesynopsis{Encode and decode data and streams.}\moduleauthor{Marc-Andre Lemburg}{mal@lemburg.com}\sectionauthor{Marc-Andre Lemburg}{mal@lemburg.com}\sectionauthor{Martin v. L\"owis}{martin@v.loewis.de} \index{Unicode}\index{Codecs}\indexii{Codecs}{encode}\indexii{Codecs}{decode}\index{streams}\indexii{stackable}{streams}  This module defines base classes for standard Python codecs (encodersand decoders) and provides access to the internal Python codecregistry which manages the codec and error handling lookup process. It defines the following functions: \begin{funcdesc}{register}{search_function}Register a codec search function. Search functions are expected totake one argument, the encoding name in all lower case letters, andreturn a \class{CodecInfo} object having the following attributes: \begin{itemize}  \item \code{name} The name of the encoding;  \item \code{encoder} The stateless encoding function;  \item \code{decoder} The stateless decoding function;  \item \code{incrementalencoder} An incremental encoder class or factory function;  \item \code{incrementaldecoder} An incremental decoder class or factory function;  \item \code{streamwriter} A stream writer class or factory function;  \item \code{streamreader} A stream reader class or factory function.\end{itemize} The various functions or classes take the following arguments:   \var{encoder} and \var{decoder}: These must be functions or methods  which have the same interface as the  \method{encode()}/\method{decode()} methods of Codec instances (see  Codec Interface). The functions/methods are expected to work in a  stateless mode.   \var{incrementalencoder} and \var{incrementalencoder}: These have to be  factory functions providing the following interface:         \code{factory(\var{errors}='strict')}   The factory functions must return objects providing the interfaces  defined by the base classes \class{IncrementalEncoder} and  \class{IncrementalEncoder}, respectively. Incremental codecs can maintain  state.   \var{streamreader} and \var{streamwriter}: These have to be  factory functions providing the following interface:         \code{factory(\var{stream}, \var{errors}='strict')}   The factory functions must return objects providing the interfaces  defined by the base classes \class{StreamWriter} and  \class{StreamReader}, respectively. Stream codecs can maintain  state.   Possible values for errors are \code{'strict'} (raise an exception  in case of an encoding error), \code{'replace'} (replace malformed  data with a suitable replacement marker, such as \character{?}),  \code{'ignore'} (ignore malformed data and continue without further  notice), \code{'xmlcharrefreplace'} (replace with the appropriate XML  character reference (for encoding only)) and \code{'backslashreplace'}  (replace with backslashed escape sequences (for encoding only)) as  well as any other error handling name defined via  \function{register_error()}. In case a search function cannot find a given encoding, it shouldreturn \code{None}.\end{funcdesc} \begin{funcdesc}{lookup}{encoding}Looks up the codec info in the Python codec registry and returns a\class{CodecInfo} object as defined above. Encodings are first looked up in the registry's cache. If not found,the list of registered search functions is scanned. If no \class{CodecInfo}object is found, a \exception{LookupError} is raised. Otherwise, the\class{CodecInfo} object is stored in the cache and returned to the caller.\end{funcdesc} To simplify access to the various codecs, the module provides theseadditional functions which use \function{lookup()} for the codeclookup: \begin{funcdesc}{getencoder}{encoding}Look up the codec for the given encoding and return its encoderfunction. Raises a \exception{LookupError} in case the encoding cannot be found.\end{funcdesc} \begin{funcdesc}{getdecoder}{encoding}Look up the codec for the given encoding and return its decoderfunction. Raises a \exception{LookupError} in case the encoding cannot be found.\end{funcdesc} \begin{funcdesc}{getincrementalencoder}{encoding}Look up the codec for the given encoding and return its incremental encoderclass or factory function. Raises a \exception{LookupError} in case the encoding cannot be found or thecodec doesn't support an incremental encoder.\versionadded{2.5}\end{funcdesc} \begin{funcdesc}{getincrementaldecoder}{encoding}Look up the codec for the given encoding and return its incremental decoderclass or factory function. Raises a \exception{LookupError} in case the encoding cannot be found or thecodec doesn't support an incremental decoder.\versionadded{2.5}\end{funcdesc} \begin{funcdesc}{getreader}{encoding}Look up the codec for the given encoding and return its StreamReaderclass or factory function. Raises a \exception{LookupError} in case the encoding cannot be found.\end{funcdesc} \begin{funcdesc}{getwriter}{encoding}Look up the codec for the given encoding and return its StreamWriterclass or factory function. Raises a \exception{LookupError} in case the encoding cannot be found.\end{funcdesc} \begin{funcdesc}{register_error}{name, error_handler}Register the error handling function \var{error_handler} under thename \var{name}. \var{error_handler} will be called during encodingand decoding in case of an error, when \var{name} is specified as theerrors parameter. For encoding \var{error_handler} will be called with a\exception{UnicodeEncodeError} instance, which contains information aboutthe location of the error. The error handler must either raise this ora different exception or return a tuple with a replacement for theunencodable part of the input and a position where encoding shouldcontinue. The encoder will encode the replacement and continue encodingthe original input at the specified position. Negative position valueswill be treated as being relative to the end of the input string. If theresulting position is out of bound an \exception{IndexError} will be raised. Decoding and translating works similar, except \exception{UnicodeDecodeError}or \exception{UnicodeTranslateError} will be passed to the handler andthat the replacement from the error handler will be put into the outputdirectly.\end{funcdesc} \begin{funcdesc}{lookup_error}{name}Return the error handler previously registered under the name \var{name}. Raises a \exception{LookupError} in case the handler cannot be found.\end{funcdesc} \begin{funcdesc}{strict_errors}{exception}Implements the \code{strict} error handling.\end{funcdesc} \begin{funcdesc}{replace_errors}{exception}Implements the \code{replace} error handling.\end{funcdesc} \begin{funcdesc}{ignore_errors}{exception}Implements the \code{ignore} error handling.\end{funcdesc} \begin{funcdesc}{xmlcharrefreplace_errors_errors}{exception}Implements the \code{xmlcharrefreplace} error handling.\end{funcdesc} \begin{funcdesc}{backslashreplace_errors_errors}{exception}Implements the \code{backslashreplace} error handling.\end{funcdesc} To simplify working with encoded files or stream, the modulealso defines these utility functions: \begin{funcdesc}{open}{filename, mode\optional{, encoding\optional{,                       errors\optional{, buffering}}}}Open an encoded file using the given \var{mode} and returna wrapped version providing transparent encoding/decoding. \note{The wrapped version will only accept the object formatdefined by the codecs, i.e.\ Unicode objects for most built-incodecs.  Output is also codec-dependent and will usually be Unicode aswell.} \var{encoding} specifies the encoding which is to be used for thefile. \var{errors} may be given to define the error handling. It defaultsto \code{'strict'} which causes a \exception{ValueError} to be raisedin case an encoding error occurs. \var{buffering} has the same meaning as for the built-in\function{open()} function.  It defaults to line buffered.\end{funcdesc} \begin{funcdesc}{EncodedFile}{file, input\optional{,                              output\optional{, errors}}}Return a wrapped version of file which provides transparentencoding translation. Strings written to the wrapped file are interpreted according to thegiven \var{input} encoding and then written to the original file asstrings using the \var{output} encoding. The intermediate encoding willusually be Unicode but depends on the specified codecs. If \var{output} is not given, it defaults to \var{input}. \var{errors} may be given to define the error handling. It defaults to\code{'strict'}, which causes \exception{ValueError} to be raised in casean encoding error occurs.\end{funcdesc} \begin{funcdesc}{iterencode}{iterable, encoding\optional{, errors}}Uses an incremental encoder to iteratively encode the input provided by\var{iterable}. This function is a generator. \var{errors} (as well asany other keyword argument) is passed through to the incremental encoder.\versionadded{2.5}\end{funcdesc} \begin{funcdesc}{iterdecode}{iterable, encoding\optional{, errors}}Uses an incremental decoder to iteratively decode the input provided by\var{iterable}. This function is a generator. \var{errors} (as well asany other keyword argument) is passed through to the incremental encoder.\versionadded{2.5}\end{funcdesc} The module also provides the following constants which are usefulfor reading and writing to platform dependent files: \begin{datadesc}{BOM}\dataline{BOM_BE}\dataline{BOM_LE}\dataline{BOM_UTF8}\dataline{BOM_UTF16}\dataline{BOM_UTF16_BE}\dataline{BOM_UTF16_LE}\dataline{BOM_UTF32}\dataline{BOM_UTF32_BE}\dataline{BOM_UTF32_LE}These constants define various encodings of the Unicode byte order mark(BOM) used in UTF-16 and UTF-32 data streams to indicate the byte orderused in the stream or file and in UTF-8 as a Unicode signature.\constant{BOM_UTF16} is either \constant{BOM_UTF16_BE} or\constant{BOM_UTF16_LE} depending on the platform's native byte order,\constant{BOM} is an alias for \constant{BOM_UTF16}, \constant{BOM_LE}for \constant{BOM_UTF16_LE} and \constant{BOM_BE} for \constant{BOM_UTF16_BE}.The others represent the BOM in UTF-8 and UTF-32 encodings.\end{datadesc}  \subsection{Codec Base Classes \label{codec-base-classes}} The \module{codecs} module defines a set of base classes which define theinterface and can also be used to easily write you own codecs for usein Python. Each codec has to define four interfaces to make it usable as codec inPython: stateless encoder, stateless decoder, stream reader and streamwriter. The stream reader and writers typically reuse the statelessencoder/decoder to implement the file protocols. The \class{Codec} class defines the interface for statelessencoders/decoders. To simplify and standardize error handling, the \method{encode()} and\method{decode()} methods may implement different error handlingschemes by providing the \var{errors} string argument.  The followingstring values are defined and implemented by all standard Pythoncodecs: \begin{tableii}{l|l}{code}{Value}{Meaning}  \lineii{'strict'}{Raise \exception{UnicodeError} (or a subclass);                    this is the default.}  \lineii{'ignore'}{Ignore the character and continue with the next.}  \lineii{'replace'}{Replace with a suitable replacement character;                     Python will use the official U+FFFD REPLACEMENT                     CHARACTER for the built-in Unicode codecs on                     decoding and '?' on encoding.}  \lineii{'xmlcharrefreplace'}{Replace with the appropriate XML                     character reference (only for encoding).}  \lineii{'backslashreplace'}{Replace with backslashed escape sequences                     (only for encoding).}\end{tableii} The set of allowed values can be extended via \method{register_error}.  \subsubsection{Codec Objects \label{codec-objects}} The \class{Codec} class defines these methods which also define thefunction interfaces of the stateless encoder and decoder: \begin{methoddesc}{encode}{input\optional{, errors}}  Encodes the object \var{input} and returns a tuple (output object,  length consumed).  While codecs are not restricted to use with Unicode, in  a Unicode context, encoding converts a Unicode object to a plain string  using a particular character set encoding (e.g., \code{cp1252} or  \code{iso-8859-1}).   \var{errors} defines the error handling to apply. It defaults to  \code{'strict'} handling.   The method may not store state in the \class{Codec} instance. Use  \class{StreamCodec} for codecs which have to keep state in order to  make encoding/decoding efficient.   The encoder must be able to handle zero length input and return an  empty object of the output object type in this situation.\end{methoddesc} \begin{methoddesc}{decode}{input\optional{, errors}}  Decodes the object \var{input} and returns a tuple (output object,  length consumed).  In a Unicode context, decoding converts a plain string  encoded using a particular character set encoding to a Unicode object.   \var{input} must be an object which provides the \code{bf_getreadbuf}  buffer slot.  Python strings, buffer objects and memory mapped files  are examples of objects providing this slot.   \var{errors} defines the error handling to apply. It defaults to  \code{'strict'} handling.   The method may not store state in the \class{Codec} instance. Use  \class{StreamCodec} for codecs which have to keep state in order to  make encoding/decoding efficient.   The decoder must be able to handle zero length input and return an  empty object of the output object type in this situation.\end{methoddesc} The \class{IncrementalEncoder} and \class{IncrementalDecoder} classes providethe basic interface for incremental encoding and decoding. Encoding/decoding theinput isn't done with one call to the stateless encoder/decoder function,but with multiple calls to the \method{encode}/\method{decode} method of theincremental encoder/decoder. The incremental encoder/decoder keeps track ofthe encoding/decoding process during method calls. The joined output of calls to the \method{encode}/\method{decode} method is thesame as if all the single inputs were joined into one, and this input wasencoded/decoded with the stateless encoder/decoder.  \subsubsection{IncrementalEncoder Objects \label{incremental-encoder-objects}} \versionadded{2.5} The \class{IncrementalEncoder} class is used for encoding an input in multiplesteps. It defines the following methods which every incremental encoder mustdefine in order to be compatible with the Python codec registry. \begin{classdesc}{IncrementalEncoder}{\optional{errors}}  Constructor for an \class{IncrementalEncoder} instance.   All incremental encoders must provide this constructor interface. They are  free to add additional keyword arguments, but only the ones defined  here are used by the Python codec registry.   The \class{IncrementalEncoder} may implement different error handling  schemes by providing the \var{errors} keyword argument. These  parameters are predefined:   \begin{itemize}    \item \code{'strict'} Raise \exception{ValueError} (or a subclass);                          this is the default.    \item \code{'ignore'} Ignore the character and continue with the next.    \item \code{'replace'} Replace with a suitable replacement character    \item \code{'xmlcharrefreplace'} Replace with the appropriate XML                     character reference    \item \code{'backslashreplace'} Replace with backslashed escape sequences.  \end{itemize}   The \var{errors} argument will be assigned to an attribute of the  same name. Assigning to this attribute makes it possible to switch  between different error handling strategies during the lifetime  of the \class{IncrementalEncoder} object.   The set of allowed values for the \var{errors} argument can  be extended with \function{register_error()}.\end{classdesc} \begin{methoddesc}{encode}{object\optional{, final}}  Encodes \var{object} (taking the current state of the encoder into account)  and returns the resulting encoded object. If this is the last call to  \method{encode} \var{final} must be true (the default is false).\end{methoddesc} \begin{methoddesc}{reset}{}  Reset the encoder to the initial state.\end{methoddesc}  \subsubsection{IncrementalDecoder Objects \label{incremental-decoder-objects}} The \class{IncrementalDecoder} class is used for decoding an input in multiplesteps. It defines the following methods which every incremental decoder mustdefine in order to be compatible with the Python codec registry. \begin{classdesc}{IncrementalDecoder}{\optional{errors}}  Constructor for an \class{IncrementalDecoder} instance.   All incremental decoders must provide this constructor interface. They are  free to add additional keyword arguments, but only the ones defined  here are used by the Python codec registry.   The \class{IncrementalDecoder} may implement different error handling  schemes by providing the \var{errors} keyword argument. These  parameters are predefined:   \begin{itemize}    \item \code{'strict'} Raise \exception{ValueError} (or a subclass);                          this is the default.    \item \code{'ignore'} Ignore the character and continue with the next.    \item \code{'replace'} Replace with a suitable replacement character.  \end{itemize}   The \var{errors} argument will be assigned to an attribute of the  same name. Assigning to this attribute makes it possible to switch  between different error handling strategies during the lifetime  of the \class{IncrementalEncoder} object.   The set of allowed values for the \var{errors} argument can  be extended with \function{register_error()}.\end{classdesc} \begin{methoddesc}{decode}{object\optional{, final}}  Decodes \var{object} (taking the current state of the decoder into account)  and returns the resulting decoded object. If this is the last call to  \method{decode} \var{final} must be true (the default is false).  If \var{final} is true the decoder must decode the input completely and must  flush all buffers. If this isn't possible (e.g. because of incomplete byte  sequences at the end of the input) it must initiate error handling just like  in the stateless case (which might raise an exception).\end{methoddesc} \begin{methoddesc}{reset}{}  Reset the decoder to the initial state.\end{methoddesc}  The \class{StreamWriter} and \class{StreamReader} classes providegeneric working interfaces which can be used to implement newencoding submodules very easily. See \module{encodings.utf_8} for anexample of how this is done.  \subsubsection{StreamWriter Objects \label{stream-writer-objects}} The \class{StreamWriter} class is a subclass of \class{Codec} anddefines the following methods which every stream writer must define inorder to be compatible with the Python codec registry. \begin{classdesc}{StreamWriter}{stream\optional{, errors}}  Constructor for a \class{StreamWriter} instance.    All stream writers must provide this constructor interface. They are  free to add additional keyword arguments, but only the ones defined  here are used by the Python codec registry.   \var{stream} must be a file-like object open for writing binary  data.   The \class{StreamWriter} may implement different error handling  schemes by providing the \var{errors} keyword argument. These  parameters are predefined:   \begin{itemize}    \item \code{'strict'} Raise \exception{ValueError} (or a subclass);                          this is the default.    \item \code{'ignore'} Ignore the character and continue with the next.    \item \code{'replace'} Replace with a suitable replacement character    \item \code{'xmlcharrefreplace'} Replace with the appropriate XML                     character reference    \item \code{'backslashreplace'} Replace with backslashed escape sequences.  \end{itemize}   The \var{errors} argument will be assigned to an attribute of the  same name. Assigning to this attribute makes it possible to switch  between different error handling strategies during the lifetime  of the \class{StreamWriter} object.   The set of allowed values for the \var{errors} argument can  be extended with \function{register_error()}.\end{classdesc}

Lines 1–500 of 1348. Load more