\section{\module{HTMLParser} ---         Simple HTML and XHTML parser} \declaremodule{standard}{HTMLParser}\modulesynopsis{A simple parser that can handle HTML and XHTML.} \versionadded{2.2} This module defines a class \class{HTMLParser} which serves as thebasis for parsing text files formatted in HTML\index{HTML} (HyperTextMark-up Language) and XHTML.\index{XHTML}  Unlike the parser in\refmodule{htmllib}, this parser is not based on the SGML parser in\refmodule{sgmllib}.  \begin{classdesc}{HTMLParser}{}The \class{HTMLParser} class is instantiated without arguments. An HTMLParser instance is fed HTML data and calls handler functionswhen tags begin and end.  The \class{HTMLParser} class is meant to beoverridden by the user to provide a desired behavior. Unlike the parser in \refmodule{htmllib}, this parser does not checkthat end tags match start tags or call the end-tag handler forelements which are closed implicitly by closing an outer element.\end{classdesc} An exception is defined as well: \begin{excdesc}{HTMLParseError}Exception raised by the \class{HTMLParser} class when it encounters anerror while parsing.  This exception provides three attributes:\member{msg} is a brief message explaining the error, \member{lineno}is the number of the line on which the broken construct was detected,and \member{offset} is the number of characters into the line at whichthe construct starts.\end{excdesc}  \class{HTMLParser} instances have the following methods: \begin{methoddesc}{reset}{}Reset the instance.  Loses all unprocessed data.  This is calledimplicitly at instantiation time.\end{methoddesc} \begin{methoddesc}{feed}{data}Feed some text to the parser.  It is processed insofar as it consistsof complete elements; incomplete data is buffered until more data isfed or \method{close()} is called.\end{methoddesc} \begin{methoddesc}{close}{}Force processing of all buffered data as if it were followed by anend-of-file mark.  This method may be redefined by a derived class todefine additional processing at the end of the input, but theredefined version should always call the \class{HTMLParser} base classmethod \method{close()}.\end{methoddesc} \begin{methoddesc}{getpos}{}Return current line number and offset.\end{methoddesc} \begin{methoddesc}{get_starttag_text}{}Return the text of the most recently opened start tag.  This shouldnot normally be needed for structured processing, but may be useful indealing with HTML ``as deployed'' or for re-generating input withminimal changes (whitespace between attributes can be preserved,etc.).\end{methoddesc} \begin{methoddesc}{handle_starttag}{tag, attrs} This method is called to handle the start of a tag.  It is intended tobe overridden by a derived class; the base class implementation doesnothing.   The \var{tag} argument is the name of the tag converted tolower case.  The \var{attrs} argument is a list of \code{(\var{name},\var{value})} pairs containing the attributes found inside the tag's\code{<>} brackets.  The \var{name} will be translated to lower caseand double quotes and backslashes in the \var{value} have beeninterpreted.  For instance, for the tag \code{<AHREF="http://www.cwi.nl/">}, this method would be called as\samp{handle_starttag('a', [('href', 'http://www.cwi.nl/')])}.\end{methoddesc} \begin{methoddesc}{handle_startendtag}{tag, attrs}Similar to \method{handle_starttag()}, but called when the parserencounters an XHTML-style empty tag (\code{<a .../>}).  This methodmay be overridden by subclasses which require this particular lexicalinformation; the default implementation simple calls\method{handle_starttag()} and \method{handle_endtag()}.\end{methoddesc} \begin{methoddesc}{handle_endtag}{tag}This method is called to handle the end tag of an element.  It isintended to be overridden by a derived class; the base classimplementation does nothing.  The \var{tag} argument is the name ofthe tag converted to lower case.\end{methoddesc} \begin{methoddesc}{handle_data}{data}This method is called to process arbitrary data.  It is intended to beoverridden by a derived class; the base class implementation doesnothing.\end{methoddesc} \begin{methoddesc}{handle_charref}{name} This method is called toprocess a character reference of the form \samp{\&\#\var{ref};}.  Itis intended to be overridden by a derived class; the base classimplementation does nothing.  \end{methoddesc} \begin{methoddesc}{handle_entityref}{name} This method is called to process a general entity reference of theform \samp{\&\var{name};} where \var{name} is an general entityreference.  It is intended to be overridden by a derived class; thebase class implementation does nothing.\end{methoddesc} \begin{methoddesc}{handle_comment}{data}This method is called when a comment is encountered.  The\var{comment} argument is a string containing the text between the\samp{--} and \samp{--} delimiters, but not the delimitersthemselves.  For example, the comment \samp{<!--text-->} willcause this method to be called with the argument \code{'text'}.  It isintended to be overridden by a derived class; the base classimplementation does nothing.\end{methoddesc} \begin{methoddesc}{handle_decl}{decl}Method called when an SGML declaration is read by the parser.  The\var{decl} parameter will be the entire contents of the declarationinside the \code{<!}...\code{>} markup.  It is intended to be overriddenby a derived class; the base class implementation does nothing.\end{methoddesc} \begin{methoddesc}{handle_pi}{data}Method called when a processing instruction is encountered.  The\var{data} parameter will contain the entire processing instruction.For example, for the processing instruction \code{<?proc color='red'>},this method would be called as \code{handle_pi("proc color='red'")}.  Itis intended to be overridden by a derived class; the base classimplementation does nothing. \note{The \class{HTMLParser} class uses the SGML syntactic rules forprocessing instructions.  An XHTML processing instruction using thetrailing \character{?} will cause the \character{?} to be included in\var{data}.}\end{methoddesc}  \subsection{Example HTML Parser Application \label{htmlparser-example}} As a basic example, below is a very basic HTML parser that uses the\class{HTMLParser} class to print out tags as they are encountered: \begin{verbatim}from HTMLParser import HTMLParser class MyHTMLParser(HTMLParser):     def handle_starttag(self, tag, attrs):        print "Encountered the beginning of a %s tag" % tag     def handle_endtag(self, tag):        print "Encountered the end of a %s tag" % tag\end{verbatim}