\chapter{Lexical analysis\label{lexical}} A Python program is read by a \emph{parser}.  Input to the parser is astream of \emph{tokens}, generated by the \emph{lexical analyzer}.  Thischapter describes how the lexical analyzer breaks a file into tokens.\index{lexical analysis}\index{parser}\index{token} Python uses the 7-bit \ASCII{} character set for program text.\versionadded[An encoding declaration can be used to indicate that string literals and comments use an encoding different from ASCII]{2.3}For compatibility with older versions, Python only warns if it finds8-bit characters; those warnings should be corrected by either declaringan explicit encoding, or using escape sequences if those bytes are binarydata, instead of characters.  The run-time character set depends on the I/O devices connected to theprogram but is generally a superset of \ASCII. \strong{Future compatibility note:} It may be tempting to assume that thecharacter set for 8-bit characters is ISO Latin-1 (an \ASCII{}superset that covers most western languages that use the Latinalphabet), but it is possible that in the future Unicode text editorswill become common.  These generally use the UTF-8 encoding, which isalso an \ASCII{} superset, but with very different use for thecharacters with ordinals 128-255.  While there is no consensus on thissubject yet, it is unwise to assume either Latin-1 or UTF-8, eventhough the current implementation appears to favor Latin-1.  Thisapplies both to the source character set and the run-time characterset.  \section{Line structure\label{line-structure}} A Python program is divided into a number of \emph{logical lines}.\index{line structure}  \subsection{Logical lines\label{logical}} The end ofa logical line is represented by the token NEWLINE.  Statements cannotcross logical line boundaries except where NEWLINE is allowed by thesyntax (e.g., between statements in compound statements).A logical line is constructed from one or more \emph{physical lines}by following the explicit or implicit \emph{line joining} rules.\index{logical line}\index{physical line}\index{line joining}\index{NEWLINE token}  \subsection{Physical lines\label{physical}} A physical line is a sequence of characters terminated by an end-of-linesequence.  In source files, any of the standard platform linetermination sequences can be used - the \UNIX{} form using \ASCII{} LF(linefeed), the Windows form using the \ASCII{} sequence CR LF (returnfollowed by linefeed), or the Macintosh form using the \ASCII{} CR(return) character.  All of these forms can be used equally, regardlessof platform. When embedding Python, source code strings should be passed to PythonAPIs using the standard C conventions for newline characters (the\code{\e n} character, representing \ASCII{} LF, is the lineterminator).  \subsection{Comments\label{comments}} A comment starts with a hash character (\code{\#}) that is not part ofa string literal, and ends at the end of the physical line.  A commentsignifies the end of the logical line unless the implicit line joiningrules are invoked.Comments are ignored by the syntax; they are not tokens.\index{comment}\index{hash character}  \subsection{Encoding declarations\label{encodings}}\index{source character set}\index{encodings} If a comment in the first or second line of the Python script matchesthe regular expression \regexp{coding[=:]\e s*([-\e w.]+)}, this comment isprocessed as an encoding declaration; the first group of thisexpression names the encoding of the source code file. The recommendedforms of this expression are \begin{verbatim}# -*- coding: <encoding-name> -*-\end{verbatim} which is recognized also by GNU Emacs, and \begin{verbatim}# vim:fileencoding=<encoding-name>\end{verbatim} which is recognized by Bram Moolenaar's VIM. In addition, if the firstbytes of the file are the UTF-8 byte-order mark(\code{'\e xef\e xbb\e xbf'}), the declared file encoding is UTF-8(this is supported, among others, by Microsoft's \program{notepad}). If an encoding is declared, the encoding name must be recognized byPython. % XXX there should be a list of supported encodings.The encoding is used for all lexical analysis, in particular to findthe end of a string, and to interpret the contents of Unicode literals.String literals are converted to Unicode for syntactical analysis,then converted back to their original encoding before interpretationstarts. The encoding declaration must appear on a line of its own. \subsection{Explicit line joining\label{explicit-joining}} Two or more physical lines may be joined into logical lines usingbackslash characters (\code{\e}), as follows: when a physical line endsin a backslash that is not part of a string literal or comment, it isjoined with the following forming a single logical line, deleting thebackslash and the following end-of-line character.  For example:\index{physical line}\index{line joining}\index{line continuation}\index{backslash character}%\begin{verbatim}if 1900 < year < 2100 and 1 <= month <= 12 \   and 1 <= day <= 31 and 0 <= hour < 24 \   and 0 <= minute < 60 and 0 <= second < 60:   # Looks like a valid date        return 1\end{verbatim} A line ending in a backslash cannot carry a comment.  A backslash doesnot continue a comment.  A backslash does not continue a token exceptfor string literals (i.e., tokens other than string literals cannot besplit across physical lines using a backslash).  A backslash isillegal elsewhere on a line outside a string literal.  \subsection{Implicit line joining\label{implicit-joining}} Expressions in parentheses, square brackets or curly braces can besplit over more than one physical line without using backslashes.For example: \begin{verbatim}month_names = ['Januari', 'Februari', 'Maart',      # These are the               'April',   'Mei',      'Juni',       # Dutch names               'Juli',    'Augustus', 'September',  # for the months               'Oktober', 'November', 'December']   # of the year\end{verbatim} Implicitly continued lines can carry comments.  The indentation of thecontinuation lines is not important.  Blank continuation lines areallowed.  There is no NEWLINE token between implicit continuationlines.  Implicitly continued lines can also occur within triple-quotedstrings (see below); in that case they cannot carry comments.  \subsection{Blank lines \label{blank-lines}} \index{blank line}A logical line that contains only spaces, tabs, formfeeds and possiblya comment, is ignored (i.e., no NEWLINE token is generated).  Duringinteractive input of statements, handling of a blank line may differdepending on the implementation of the read-eval-print loop.  In thestandard implementation, an entirely blank logical line (i.e.\ onecontaining not even whitespace or a comment) terminates a multi-linestatement.  \subsection{Indentation\label{indentation}} Leading whitespace (spaces and tabs) at the beginning of a logicalline is used to compute the indentation level of the line, which inturn is used to determine the grouping of statements.\index{indentation}\index{whitespace}\index{leading whitespace}\index{space}\index{tab}\index{grouping}\index{statement grouping} First, tabs are replaced (from left to right) by one to eight spacessuch that the total number of characters up to and including thereplacement is a multiple ofeight (this is intended to be the same rule as used by \UNIX).  Thetotal number of spaces preceding the first non-blank character thendetermines the line's indentation.  Indentation cannot be split overmultiple physical lines using backslashes; the whitespace up to thefirst backslash determines the indentation. \strong{Cross-platform compatibility note:} because of the nature oftext editors on non-UNIX platforms, it is unwise to use a mixture ofspaces and tabs for the indentation in a single source file.  Itshould also be noted that different platforms may explicitly limit themaximum indentation level. A formfeed character may be present at the start of the line; it willbe ignored for the indentation calculations above.  Formfeedcharacters occurring elsewhere in the leading whitespace have anundefined effect (for instance, they may reset the space count tozero). The indentation levels of consecutive lines are used to generateINDENT and DEDENT tokens, using a stack, as follows.\index{INDENT token}\index{DEDENT token} Before the first line of the file is read, a single zero is pushed onthe stack; this will never be popped off again.  The numbers pushed onthe stack will always be strictly increasing from bottom to top.  Atthe beginning of each logical line, the line's indentation level iscompared to the top of the stack.  If it is equal, nothing happens.If it is larger, it is pushed on the stack, and one INDENT token isgenerated.  If it is smaller, it \emph{must} be one of the numbersoccurring on the stack; all numbers on the stack that are larger arepopped off, and for each number popped off a DEDENT token isgenerated.  At the end of the file, a DEDENT token is generated foreach number remaining on the stack that is larger than zero. Here is an example of a correctly (though confusingly) indented pieceof Python code: \begin{verbatim}def perm(l):        # Compute the list of all permutations of l    if len(l) <= 1:                  return [l]    r = []    for i in range(len(l)):             s = l[:i] + l[i+1:]             p = perm(s)             for x in p:              r.append(l[i:i+1] + x)    return r\end{verbatim} The following example shows various indentation errors: \begin{verbatim} def perm(l):                       # error: first line indentedfor i in range(len(l)):             # error: not indented    s = l[:i] + l[i+1:]        p = perm(l[:i] + l[i+1:])   # error: unexpected indent        for x in p:                r.append(l[i:i+1] + x)            return r                # error: inconsistent dedent\end{verbatim} (Actually, the first three errors are detected by the parser; only thelast error is found by the lexical analyzer --- the indentation of\code{return r} does not match a level popped off the stack.)  \subsection{Whitespace between tokens\label{whitespace}} Except at the beginning of a logical line or in string literals, thewhitespace characters space, tab and formfeed can be usedinterchangeably to separate tokens.  Whitespace is needed between twotokens only if their concatenation could otherwise be interpreted as adifferent token (e.g., ab is one token, but a b is two tokens).  \section{Other tokens\label{other-tokens}} Besides NEWLINE, INDENT and DEDENT, the following categories of tokensexist: \emph{identifiers}, \emph{keywords}, \emph{literals},\emph{operators}, and \emph{delimiters}.Whitespace characters (other than line terminators, discussed earlier)are not tokens, but serve to delimit tokens.Whereambiguity exists, a token comprises the longest possible string thatforms a legal token, when read from left to right.  \section{Identifiers and keywords\label{identifiers}} Identifiers (also referred to as \emph{names}) are described by the followinglexical definitions:\index{identifier}\index{name} \begin{productionlist}  \production{identifier}             {(\token{letter}|"_") (\token{letter} | \token{digit} | "_")*}  \production{letter}             {\token{lowercase} | \token{uppercase}}  \production{lowercase}             {"a"..."z"}  \production{uppercase}             {"A"..."Z"}  \production{digit}             {"0"..."9"}\end{productionlist} Identifiers are unlimited in length.  Case is significant.  \subsection{Keywords\label{keywords}} The following identifiers are used as reserved words, or\emph{keywords} of the language, and cannot be used as ordinaryidentifiers.  They must be spelled exactly as written here:%\index{keyword}%\index{reserved word} \begin{verbatim}and       del       from      not       while    as        elif      global    or        with     assert    else      if        pass      yield    break     except    import    print              class     exec      in        raise              continue  finally   is        return             def       for       lambda    try \end{verbatim} % When adding keywords, use reswords.py for reformatting \versionchanged[\constant{None} became a constant and is nowrecognized by the compiler as a name for the built-in object\constant{None}.  Although it is not a keyword, you cannot assigna different object to it]{2.4} \versionchanged[Both \keyword{as} and \keyword{with} are only recognizedwhen the \code{with_statement} future feature has been enabled.It will always be enabled in Python 2.6.  See section~\ref{with} fordetails.  Note that using \keyword{as} and \keyword{with} as identifierswill always issue a warning, even when the \code{with_statement} futuredirective is not in effect]{2.5}  \subsection{Reserved classes of identifiers\label{id-classes}} Certain classes of identifiers (besides keywords) have specialmeanings.  These classes are identified by the patterns of leading andtrailing underscore characters: \begin{description} \item[\code{_*}]  Not imported by \samp{from \var{module} import *}.  The special  identifier \samp{_} is used in the interactive interpreter to store  the result of the last evaluation; it is stored in the  \module{__builtin__} module.  When not in interactive mode, \samp{_}  has no special meaning and is not defined.  See section~\ref{import}, ``The \keyword{import} statement.''   \note{The name \samp{_} is often used in conjunction with  internationalization; refer to the documentation for the  \ulink{\module{gettext} module}{../lib/module-gettext.html} for more  information on this convention.} \item[\code{__*__}]  System-defined names.  These names are defined by the interpreter  and its implementation (including the standard library);  applications should not expect to define additional names using this  convention.  The set of names of this class defined by Python may be  extended in future versions.  See section~\ref{specialnames}, ``Special method names.'' \item[\code{__*}]  Class-private names.  Names in this category, when used within the  context of a class definition, are re-written to use a mangled form  to help avoid name clashes between ``private'' attributes of base  and derived classes.  See section~\ref{atom-identifiers}, ``Identifiers (Names).'' \end{description}  \section{Literals\label{literals}} Literals are notations for constant values of some built-in types.\index{literal}\index{constant}  \subsection{String literals\label{strings}} String literals are described by the following lexical definitions:\index{string literal} \index{ASCII@\ASCII}\begin{productionlist}  \production{stringliteral}             {[\token{stringprefix}](\token{shortstring} | \token{longstring})}  \production{stringprefix}             {"r" | "u" | "ur" | "R" | "U" | "UR" | "Ur" | "uR"}  \production{shortstring}             {"'" \token{shortstringitem}* "'"              | '"' \token{shortstringitem}* '"'}  \production{longstring}             {"'''" \token{longstringitem}* "'''"}  \productioncont{| '"""' \token{longstringitem}* '"""'}  \production{shortstringitem}             {\token{shortstringchar} | \token{escapeseq}}  \production{longstringitem}             {\token{longstringchar} | \token{escapeseq}}  \production{shortstringchar}             {<any source character except "\e" or newline or the quote>}  \production{longstringchar}             {<any source character except "\e">}  \production{escapeseq}             {"\e" <any ASCII character>}\end{productionlist} One syntactic restriction not indicated by these productions is thatwhitespace is not allowed between the \grammartoken{stringprefix} andthe rest of the string literal. The source character set is definedby the encoding declaration; it is \ASCII{} if no encoding declarationis given in the source file; see section~\ref{encodings}. \index{triple-quoted string}\index{Unicode Consortium}\index{string!Unicode}In plain English: String literals can be enclosed in matching singlequotes (\code{'}) or double quotes (\code{"}).  They can also beenclosed in matching groups of three single or double quotes (theseare generally referred to as \emph{triple-quoted strings}).  Thebackslash (\code{\e}) character is used to escape characters thatotherwise have a special meaning, such as newline, backslash itself,or the quote character.  String literals may optionally be prefixedwith a letter \character{r} or \character{R}; such strings are called\dfn{raw strings}\index{raw string} and use different rules for interpretingbackslash escape sequences.  A prefix of \character{u} or \character{U}makes the string a Unicode string.  Unicode strings use the Unicode characterset as defined by the Unicode Consortium and ISO~10646.  Some additionalescape sequences, described below, are available in Unicode strings.The two prefix characters may be combined; in this case, \character{u} mustappear before \character{r}. In triple-quoted strings,unescaped newlines and quotes are allowed (and are retained), exceptthat three unescaped quotes in a row terminate the string.  (A``quote'' is the character used to open the string, i.e. either\code{'} or \code{"}.) Unless an \character{r} or \character{R} prefix is present, escapesequences in strings are interpreted according to rules similarto those used by Standard C.  The recognized escape sequences are:\index{physical line}\index{escape sequence}\index{Standard C}\index{C} \begin{tableiii}{l|l|c}{code}{Escape Sequence}{Meaning}{Notes}\lineiii{\e\var{newline}} {Ignored}{}\lineiii{\e\e}	{Backslash (\code{\e})}{}\lineiii{\e'}	{Single quote (\code{'})}{}\lineiii{\e"}	{Double quote (\code{"})}{}\lineiii{\e a}	{\ASCII{} Bell (BEL)}{}\lineiii{\e b}	{\ASCII{} Backspace (BS)}{}\lineiii{\e f}	{\ASCII{} Formfeed (FF)}{}\lineiii{\e n}	{\ASCII{} Linefeed (LF)}{}\lineiii{\e N\{\var{name}\}}        {Character named \var{name} in the Unicode database (Unicode only)}{}\lineiii{\e r}	{\ASCII{} Carriage Return (CR)}{}\lineiii{\e t}	{\ASCII{} Horizontal Tab (TAB)}{}\lineiii{\e u\var{xxxx}}        {Character with 16-bit hex value \var{xxxx} (Unicode only)}{(1)}\lineiii{\e U\var{xxxxxxxx}}        {Character with 32-bit hex value \var{xxxxxxxx} (Unicode only)}{(2)}\lineiii{\e v}	{\ASCII{} Vertical Tab (VT)}{}\lineiii{\e\var{ooo}} {Character with octal value \var{ooo}}{(3,5)}\lineiii{\e x\var{hh}} {Character with hex value \var{hh}}{(4,5)}\end{tableiii}\index{ASCII@\ASCII} \noindentNotes: \begin{itemize}\item[(1)]  Individual code units which form parts of a surrogate pair can be  encoded using this escape sequence.\item[(2)]  Any Unicode character can be encoded this way, but characters  outside the Basic Multilingual Plane (BMP) will be encoded using a  surrogate pair if Python is compiled to use 16-bit code units (the  default).  Individual code units which form parts of a surrogate  pair can be encoded using this escape sequence.\item[(3)]  As in Standard C, up to three octal digits are accepted.\item[(4)]  Unlike in Standard C, at most two hex digits are accepted.\item[(5)]  In a string literal, hexadecimal and octal escapes denote the  byte with the given value; it is not necessary that the byte  encodes a character in the source character set. In a Unicode  literal, these escapes denote a Unicode character with the given  value.\end{itemize}  Unlike Standard \index{unrecognized escape sequence}C,all unrecognized escape sequences are left in the string unchanged,i.e., \emph{the backslash is left in the string}.  (This behavior is

Lines 1–500 of 731. Load more