Python and HTML Processing

h1. Links

• [Python Structured Markup Processing Tools|http://docs.python.org/lib/markup.html]
• [1]
• [2]
• [3]
• [Python Parsing Tools|http://nedbatchelder.com/text/python-parsers.html]
• [Python REGEX|http://www.amk.ca/python/howto/regex/]
• [Parser|http://www.crummy.com/software/BeautifulSoup/]
• [HTML Processing|http://www.diveintopython.org/html_processing/index.html]
• [XML Processing|http://www.diveintopython.org/xml_processing/index.html]
• [Wrestling HTML - uTidyLib|http://www.xml.com/pub/a/2004/09/08/pyxml.html]

h2. Some ASPN Examples

• [Simple Application of htmllib for Parsing HTML|http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/135005]
• [HTML Scraper|http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/286269]

h1. URLLIB Tutorial: urllib2 - The Missing Manual

HOWTO Fetch Internet Resources with Python

urllib2 Tutorial

• [4]Introduction
• [5]Fetching URLs
  • [6]Data
  • [7]Headers
• [8]Handling Exceptions
  • [9]URLError
  • [10]HTTPError
    • [11]Error Codes
  • [12]Wrapping it Up
    • [13]Number 1
    • [14]Number 2
• [15]info and geturl
• [16]Openers and Handlers
• [17]Basic Authentication
• [18]Proxies
• [19]Sockets and Layers
• [20]Footnotes

[21]Introduction

urllib2 is a Python module for fetching URLs (Uniform Resource Locators). It offers a very simple interface, in the form of the urlopen function. This is capable of fetching URLs using a variety of different protocols. It also offers a slightly more complex interface for handling common situations - like basic authentication, cookies, proxies and so on. These are provided by objects called handlers and openers.

urllib2 supports fetching URLs for many "URL schemes" (identified by the string before the ":" in URL - for example "ftp" is the URL scheme of "ftp://python.org/") using their associated network protocols (e.g. FTP, HTTP). This tutorial focuses on the most common case, HTTP.

For straightforward situations urlopen is very easy to use. But as soon as you encounter errors or non-trivial cases when opening HTTP URLs, you will need some understanding of the HyperText Transfer Protocol. The most comprehensive and authoritative reference to HTTP is RFC 2616. This is a technical document and not intended to be easy to read. This HOWTO aims to illustrate using urllib2, with enough detail about HTTP to help you through. It is not intended to replace the urllib2 docs , but is supplementary to them.

[22]Fetching URLs

The simplest way to use urllib2 is as follows :

import urllib2
response = urllib2.urlopen('http://python.org/')
html = response.read()

Many uses of urllib2 will be that simple (note that instead of an 'http:' URL we could have used an URL starting with 'ftp:', 'file:', etc.). However, it's the purpose of this tutorial to explain the more complicated cases, concentrating on HTTP.

HTTP is based on requests and responses - the client makes requests and servers send responses. urllib2 mirrors this with a Request object which represents the HTTP request you are making. In its simplest form you create a Request object that specifies the URL you want to fetch. Calling urlopen with this Request object returns a response object for the URL requested. This response is a file-like object, which means you can for example call .read() on the response :

import urllib2

req = urllib2.Request('http://www.voidspace.org.uk')
response = urllib2.urlopen(req)
the_page = response.read()

Note that urllib2 makes use of the same Request interface to handle all URL schemes. For example, you can make an FTP request like so :

req = urllib2.Request('ftp://example.com/')

In the case of HTTP, there are two extra things that Request objects allow you to do: First, you can pass data to be sent to the server. Second, you can pass extra information ("metadata") about the data or the about request itself, to the server - this information is sent as HTTP "headers". Let's look at each of these in turn.

[23]Data

Sometimes you want to send data to a URL (often the URL will refer to a CGI (Common Gateway Interface) script [24][1] or other web application). With HTTP, this is often done using what's known as a POST request. This is often what your browser does when you submit a HTML form that you filled in on the web. Not all POSTs have to come from forms: you can use a POST to transmit arbitrary data to your own application. In the common case of HTML forms, the data needs to be encoded in a standard way, and then passed to the Request object as the data argument. The encoding is done using a function from the urllib library not from urllib2.

import urllib
import urllib2

url = 'http://www.someserver.com/cgi-bin/register.cgi'
values = {'name' : 'Michael Foord',
'location' : 'Northampton',
'language' : 'Python' }

data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()

Note that other encodings are sometimes required (e.g. for file upload from HTML forms - see HTML Specification, Form Submission for more details).

If you do not pass the data argument, urllib2 uses a GET request. One way in which GET and POST requests differ is that POST requests often have "side-effects": they change the state of the system in some way (for example by placing an order with the website for a hundredweight of tinned spam to be delivered to your door). Though the HTTP standard makes it clear that POSTs are intended to always cause side-effects, and GET requests never to cause side-effects, nothing prevents a GET request from having side-effects, nor a POST requests from having no side-effects. Data can also be passed in an HTTP GET request by encoding it in the URL itself.

This is done as follows.

>>> import urllib2
>>> import urllib

>>> data = {}
>>> data['name'] = 'Somebody Here'
>>> data['location'] = 'Northampton'
>>> data['language'] = 'Python'
>>> url_values = urllib.urlencode(data)
>>> print url_values
name=Somebody+Here&language=Python&location=Northampton
>>> url = 'http://www.example.com/example.cgi'
>>> full_url = url + '?' + url_values

>>> data = urllib2.open(full_url)

Notice that the full URL is created by adding a ? to the URL, followed by the encoded values.

[25]Headers

We'll discuss here one particular HTTP header, to illustrate how to add headers to your HTTP request.

Some websites [26][2] dislike being browsed by programs, or send different versions to different browsers [27][3] . By default urllib2 identifies itself as Python-urllib/x.y (where x and y are the major and minor version numbers of the Python release, e.g. Python-urllib/2.5), which may confuse the site, or just plain not work. The way a browser identifies itself is through the User-Agent header [28][4]. When you create a Request object you can pass a dictionary of headers in. The following example makes the same request as above, but identifies itself as a version of Internet Explorer [29][5].

import urllib
import urllib2

url = 'http://www.someserver.com/cgi-bin/register.cgi'
user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
values = {'name' : 'Michael Foord',
'location' : 'Northampton',
'language' : 'Python' }
headers = { 'User-Agent' : user_agent }

data = urllib.urlencode(values)
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
the_page = response.read()

The response also has two useful methods. See the section on info and geturl which comes after we have a look at what happens when things go wrong.

[30]Handling Exceptions

urlopen raises URLError when it cannot handle a response (though as usual with Python APIs, builtin exceptions such as ValueError, TypeError etc. may also be raised).

HTTPError is the subclass of URLError raised in the specific case of HTTP URLs.

[31]URLError

Often, URLError is raised because there is no network connection (no route to the specified server), or the specified server doesn't exist. In this case, the exception raised will have a 'reason' attribute, which is a tuple containing an error code and a text error message.

e.g.

>>> req = urllib2.Request('http://www.pretend_server.org')
>>> try: urllib2.urlopen(req)
>>> except URLError, e:
>>>    print e.reason
>>>

(4, 'getaddrinfo failed')

[32]HTTPError

Every HTTP response from the server contains a numeric "status code". Sometimes the status code indicates that the server is unable to fulfil the request. The default handlers will handle some of these responses for you (for example, if the response is a "redirection" that requests the client fetch the document from a different URL, urllib2 will handle that for you). For those it can't handle, urlopen will raise an HTTPError. Typical errors include '404' (page not found), '403' (request forbidden), and '401' (authentication required).

See section 10 of RFC 2616 for a reference on all the HTTP error codes.

The HTTPError instance raised will have an integer 'code' attribute, which corresponds to the error sent by the server.

[33]Error Codes

Because the default handlers handle redirects (codes in the 300 range), and codes in the 100-299 range indicate success, you will usually only see error codes in the 400-599 range.

BaseHTTPServer.BaseHTTPRequestHandler.responses is a useful dictionary of response codes in that shows all the response codes used by RFC 2616. The dictionary is reproduced here for convenience :

# Table mapping response codes to messages; entries have the
# form {code: (shortmessage, longmessage)}.
responses = {
100: ('Continue', 'Request received, please continue'),
101: ('Switching Protocols',
'Switching to new protocol; obey Upgrade header'),

200: ('OK', 'Request fulfilled, document follows'),
201: ('Created', 'Document created, URL follows'),
202: ('Accepted',
'Request accepted, processing continues off-line'),
203: ('Non-Authoritative Information', 'Request fulfilled from cache'),
204: ('No Content', 'Request fulfilled, nothing follows'),
205: ('Reset Content', 'Clear input form for further input.'),
206: ('Partial Content', 'Partial content follows.'),

300: ('Multiple Choices',
'Object has several resources -- see URI list'),
301: ('Moved Permanently', 'Object moved permanently -- see URI list'),
302: ('Found', 'Object moved temporarily -- see URI list'),
303: ('See Other', 'Object moved -- see Method and URL list'),
304: ('Not Modified',
'Document has not changed since given time'),
305: ('Use Proxy',
'You must use proxy specified in Location to access this '
'resource.'),
307: ('Temporary Redirect',
'Object moved temporarily -- see URI list'),

400: ('Bad Request',
'Bad request syntax or unsupported method'),
401: ('Unauthorized',
'No permission -- see authorization schemes'),
402: ('Payment Required',
'No payment -- see charging schemes'),
403: ('Forbidden',
'Request forbidden -- authorization will not help'),
404: ('Not Found', 'Nothing matches the given URI'),
405: ('Method Not Allowed',
'Specified method is invalid for this server.'),
406: ('Not Acceptable', 'URI not available in preferred format.'),
407: ('Proxy Authentication Required', 'You must authenticate with '
'this proxy before proceeding.'),
408: ('Request Timeout', 'Request timed out; try again later.'),
409: ('Conflict', 'Request conflict.'),
410: ('Gone',
'URI no longer exists and has been permanently removed.'),
411: ('Length Required', 'Client must specify Content-Length.'),
412: ('Precondition Failed', 'Precondition in headers is false.'),
413: ('Request Entity Too Large', 'Entity is too large.'),
414: ('Request-URI Too Long', 'URI is too long.'),
415: ('Unsupported Media Type', 'Entity body in unsupported format.'),
416: ('Requested Range Not Satisfiable',
'Cannot satisfy request range.'),
417: ('Expectation Failed',
'Expect condition could not be satisfied.'),

500: ('Internal Server Error', 'Server got itself in trouble'),
501: ('Not Implemented',
'Server does not support this operation'),
502: ('Bad Gateway', 'Invalid responses from another server/proxy.'),
503: ('Service Unavailable',
'The server cannot process the request due to a high load'),
504: ('Gateway Timeout',
'The gateway server did not receive a timely response'),
505: ('HTTP Version Not Supported', 'Cannot fulfill request.'),
}

When an error is raised the server responds by returning an HTTP error code and an error page. You can use the HTTPError instance as a response on the page returned. This means that as well as the code attribute, it also has read, geturl, and info, methods.

>>> req = urllib2.Request('http://www.python.org/fish.html')
>>> try:
>>>     urllib2.urlopen(req)
>>> except URLError, e:

>>>     print e.code
>>>     print e.read()
>>>
404

    "http://www.w3.org/TR/html4/loose.dtd">
    type="text/css"?>

...... etc...

[34]Wrapping it Up

So if you want to be prepared for HTTPError or URLError there are two basic approaches. I prefer the second approach.

[35]Number 1

from urllib2 import Request, urlopen, URLError, HTTPError
req = Request(someurl)
try:
response = urlopen(req)
except HTTPError, e:
print 'The server couldn't fulfill the request.'
print 'Error code: ', e.code
except URLError, e:
print 'We failed to reach a server.'
print 'Reason: ', e.reason
else:
# everything is fine

Note

The except HTTPError must come first, otherwise except URLError will also catch an HTTPError.

[36]Number 2

from urllib2 import Request, urlopen, URLError
req = Request(someurl)
try:
response = urlopen(req)
except URLError, e:
if hasattr(e, 'reason'):
print 'We failed to reach a server.'
print 'Reason: ', e.reason
elif hasattr(e, 'code'):
print 'The server couldn't fulfill the request.'
print 'Error code: ', e.code
else:
# everything is fine

Note

URLError is a subclass of the built-in exception IOError.

This means that you can avoid importing URLError and use :

from urllib2 import Request, urlopen
req = Request(someurl)
try:
response = urlopen(req)
except IOError, e:
if hasattr(e, 'reason'):
print 'We failed to reach a server.'
print 'Reason: ', e.reason
elif hasattr(e, 'code'):
print 'The server couldn't fulfill the request.'
print 'Error code: ', e.code
else:
# everything is fine

Under rare circumstances urllib2 can raise socket.error.

[37]info and geturl

The response returned by urlopen (or the HTTPError instance) has two useful methods info and geturl.

geturl - this returns the real URL of the page fetched. This is useful because urlopen (or the opener object used) may have followed a redirect. The URL of the page fetched may not be the same as the URL requested.

info - this returns a dictionary-like object that describes the page fetched, particularly the headers sent by the server. It is currently an httplib.HTTPMessage instance.

Typical headers include 'Content-length', 'Content-type', and so on. See the Quick Reference to HTTP Headers for a useful listing of HTTP headers with brief explanations of their meaning and use.

[38]Openers and Handlers

When you fetch a URL you use an opener (an instance of the perhaps confusingly-named urllib2.OpenerDirector). Normally we have been using the default opener - via urlopen - but you can create custom openers. Openers use handlers. All the "heavy lifting" is done by the handlers. Each handler knows how to open URLs for a particular URL scheme (http, ftp, etc.), or how to handle an aspect of URL opening, for example HTTP redirections or HTTP cookies.

You will want to create openers if you want to fetch URLs with specific handlers installed, for example to get an opener that handles cookies, or to get an opener that does not handle redirections.

To create an opener, instantiate an OpenerDirector, and then call .add_handler(some_handler_instance) repeatedly.

Alternatively, you can use build_opener, which is a convenience function for creating opener objects with a single function call. build_opener adds several handlers by default, but provides a quick way to add more and/or override the default handlers.

Other sorts of handlers you might want to can handle proxies, authentication, and other common but slightly specialised situations.

install_opener can be used to make an opener object the (global) default opener. This means that calls to urlopen will use the opener you have installed.

Opener objects have an open method, which can be called directly to fetch urls in the same way as the urlopen function: there's no need to call install_opener, except as a convenience.

[39]Basic Authentication

To illustrate creating and installing a handler we will use the HTTPBasicAuthHandler. For a more detailed discussion of this subject - including an explanation of how Basic Authentication works - see the Basic Authentication Tutorial.

When authentication is required, the server sends a header (as well as the 401 error code) requesting authentication. This specifies the authentication scheme and a 'realm'. The header looks like : Www-authenticate: SCHEME realm="REALM".

e.g.

Www-authenticate: Basic realm="cPanel Users"

The client should then retry the request with the appropriate name and password for the realm included as a header in the request. This is 'basic authentication'. In order to simplify this process we can create an instance of HTTPBasicAuthHandler and an opener to use this handler.

The HTTPBasicAuthHandler uses an object called a password manager to handle the mapping of URLs and realms to passwords and usernames. If you know what the realm is (from the authentication header sent by the server), then you can use a HTTPPasswordMgr. Frequently one doesn't care what the realm is. In that case, it is convenient to use HTTPPasswordMgrWithDefaultRealm. This allows you to specify a default username and password for a URL. This will be supplied in the absence of you providing an alternative combination for a specific realm. We indicate this by providing None as the realm argument to the add_password method.

The top-level URL is the first URL that requires authentication. URLs "deeper" than the URL you pass to .add_password() will also match.

# create a password manager
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()

# Add the username and password.
# If we knew the realm, we could use it instead of ``None``.
top_level_url = "http://example.com/foo/"
password_mgr.add_password(None, top_level_url, username, password)

handler = urllib2.HTTPBasicAuthHandler(password_mgr)

# create "opener" (OpenerDirector instance)
opener = urllib2.build_opener(handler)

# use the opener to fetch a URL
opener.open(a_url)

# Install the opener.
# Now all calls to urllib2.urlopen use our opener.
urllib2.install_opener(opener)

Note

In the above example we only supplied our HHTPBasicAuthHandler to build_opener. By default openers have the handlers for normal situations - ProxyHandler, UnknownHandler, HTTPHandler, HTTPDefaultErrorHandler, HTTPRedirectHandler, FTPHandler, FileHandler, HTTPErrorProcessor.

top_level_url is in fact either a full URL (including the 'http:' scheme component and the hostname and optionally the port number) e.g. "http://example.com/" or an "authority" (i.e. the hostname, optionally including the port number) e.g. "example.com" or "example.com:8080" (the latter example includes a port number). The authority, if present, must NOT contain the "userinfo" component - for example "joe:example.com" is not correct.

[40]Proxies

urllib2 will auto-detect your proxy settings and use those. This is through the ProxyHandler which is part of the normal handler chain. Normally that's a good thing, but there are occasions when it may not be helpful [41][6]. One way to do this is to setup our own ProxyHandler, with no proxies defined. This is done using similar steps to setting up a Basic Authentication handler :

>>> proxy_support = urllib2.ProxyHandler({})
>>> opener = urllib2.build_opener(proxy_support)
>>> urllib2.install_opener(opener)

Note

Currently urllib2 does not support fetching of https locations through a proxy. This can be a problem.

[42]Sockets and Layers

The Python support for fetching resources from the web is layered. urllib2 uses the httplib library, which in turn uses the socket library.

As of Python 2.3 you can specify how long a socket should wait for a response before timing out. This can be useful in applications which have to fetch web pages. By default the socket module has no timeout and can hang. Currently, the socket timeout is not exposed at the httplib or urllib2 levels. However, you can set the default timeout globally for all sockets using :

import socket
import urllib2

# timeout in seconds
timeout = 10
socket.setdefaulttimeout(timeout)

# this call to urllib2.urlopen now uses the default timeout
# we have set in the socket module
req = urllib2.Request('http://www.voidspace.org.uk')
response = urllib2.urlopen(req)

---

[43]Footnotes

This document was reviewed and revised by John Lee.

[44][1]

For an introduction to the CGI protocol see Writing Web Applications in Python.

[45][2]

Like Google for example. The proper way to use google from a program is to use PyGoogle of course. See Voidspace Google for some examples of using the Google API.

[46][3]

Browser sniffing is a very bad practise for website design - building sites using web standards is much more sensible. Unfortunately a lot of sites still send different versions to different browsers.

[47][4]

The user agent for MSIE 6 is 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)'

[48][5]

For details of more HTTP request headers, see Quick Reference to HTTP Headers.

[49][6]

In my case I have to use a proxy to access the internet at work. If you attempt to fetch localhost URLs through this proxy it blocks them. IE is set to use the proxy, which urllib2 picks up on. In order to test scripts with a localhost server, I have to prevent urllib2 from using the proxy.

Article Extracted from [50]>

Abstract

Various Web surfing tasks that I regularly perform could be made much easier, and less tedious, if I could only use Python to fetch the HTML pages and to process them, yielding the information I really need. In this document I attempt to describe HTML processing in Python using readily available tools and libraries.

NOTE: This document is not quite finished. I aim to include sections on using mxTidy to deal with broken HTML as well as some tips on cleaning up text retrieved from HTML resources.

Prerequisites

Depending on the methods you wish to follow in this tutorial, you need the following things:

• For the "SGML parser" method, a recent release of Python is probably enough. You can find one at the Python download page.
• For the "XML parser" method, a recent release of Python is required, along with a capable XML processing library. I recommend using libxml2dom, since it can handle badly-formed HTML documents as well as well-formed XML or XHTML documents. However, PyXML also provides support for such documents.
• For fetching Web pages over secure connections, it is important that SSL support is enabled either when building Python from source, or in any packaged distribution of Python that you might acquire. Information about this is given in the source distribution of Python, but you can download replacement socket libraries with SSL support for older versions of Python for Windows from Robin Dunn's site.

Activities

Accessing sites, downloading content, and processing such content, either to extract useful information for archiving or to use such content to navigate further into the site, require combinations of the following activities. Some activities can be chosen according to preference: whether the SGML parser or the XML parser (or parsing framework) is used depends on which style of programming seems nicer to a given developer (although one parser may seem to work better in some situations). However, technical restrictions usually dictate whether certain libraries are to be used instead of others: when handling HTTP redirects, it appears that certain Python modules are easier to use, or even more suited to handling such situations.

Fetching Web Pages

Fetching standard Web pages over HTTP is very easy with Python:

import urllib
# Get a file-like object for the Python Web site's home page.
f = urllib.urlopen("http://www.python.org")
# Read from the object, storing the page's contents in 's'.
s = f.read()
f.close()

Supplying Data

Sometimes, it is necessary to pass information to the Web server, such as information which would come from an HTML form. Of course, you need to know which fields are available in a form, but assuming that you already know this, you can supply such data in the urlopen function call:

# Search the Vaults of Parnassus for "XMLForms".
# First, encode the data.
data = urllib.urlencode({"find" : "XMLForms", "findtype" : "t"})
# Now get that file-like object again, remembering to mention the data.
f = urllib.urlopen("http://www.vex.net/parnassus/apyllo.py", data)
# Read the results back.
s = f.read()
s.close()

The above example passed data to the server as an HTTP POST request. Fortunately, the Vaults of Parnassus is happy about such requests, but this is not always the case with Web services. We can instead choose to use a different kind of request, however:

# We have the encoded data. Now get the file-like object...
f = urllib.urlopen("http://www.vex.net/parnassus/apyllo.py?" + data)
# And the rest...

The only difference is the use of a ? (question mark) character and the adding of data onto the end of the Vaults of Parnassus URL, but this constitutes an HTTP GET request, where the query (our additional data) is included in the URL itself.

Fetching Secure Web Pages

Fetching secure Web pages using HTTPS is also very easy, provided that your Python installation supports SSL:

import urllib
# Get a file-like object for a site.
f = urllib.urlopen("https://www.somesecuresite.com")
# NOTE: At the interactive Python prompt, you may be prompted for a username
# NOTE: and password here.
# Read from the object, storing the page's contents in 's'.
s = f.read()
f.close()

Including data which forms the basis of a query, as illustrated above, is also possible with URLs starting with https.

Handling Redirects

Many Web services use HTTP redirects for various straightforward or even bizarre purposes. For example, a fairly common technique employed on "high traffic" Web sites is the HTTP redirection load balancing strategy where the initial request to the publicised Web site (eg. http://www.somesite.com) is redirected to another server (eg. http://www1.somesite.com) where a user's session is handled.

Fortunately, urlopen handles redirects, at least in Python 2.1, and therefore any such redirection should be handled transparently by urlopen without your program needing to be aware that it is happening. It is possible to write code to deal with redirection yourself, and this can be done using the httplib module; however, the interfaces provided by that module are more complicated than those provided above, if somewhat more powerful.

Using the SGML Parser

Given a character string from a Web service, such as the value held by s in the above examples, how can one understand the content provided by the service in such a way that an "intelligent" response can be made? One method is by using an SGML parser, since HTML is a relation of SGML, and HTML is probably the content type most likely to be experienced when interacting with a Web service.

In the standard Python library, the sgmllib module contains an appropriate parser class called SGMLParser. Unfortunately, it is of limited use to us unless we customise its activities somehow. Fortunately, Python's object-oriented features, combined with the design of the SGMLParser class, provide a means of customising it fairly easily.

Defining a Parser Class

First of all, let us define a new class inheriting from SGMLParser with a convenience method that I find very convenient indeed:

import sgmllib

class MyParser(sgmllib.SGMLParser):
    "A simple parser class."

    def parse(self, s):
        "Parse the given string 's'."
        self.feed(s)
        self.close()

    # More to come...

What the parse method does is provide an easy way of passing some text (as a string) to the parser object. I find this nicer than having to remember calling the feed method, and since I always tend to have the entire document ready for parsing, I do not need to use feed many times - passing many pieces of text which comprise an entire document is an interesting feature of SGMLParser (and its derivatives) which could be used in other situations.

Deciding What to Remember

Of course, implementing our own customised parser is only of interest if we are looking to find things in a document. Therefore, we should aim to declare these things before we start parsing. We can do this in the __init__ method of our class:

    # Continuing from above...

    def __init__(self, verbose=0):
        "Initialise an object, passing 'verbose' to the superclass."

        sgmllib.SGMLParser.__init__(self, verbose)
        self.hyperlinks = []

    # More to come...

Here, we initialise new objects by passing information to the __init__ method of the superclass (SGMLParser); this makes sure that the underlying parser is set up properly. We also initialise an attribute called hyperlinks which will be used to record the hyperlinks found in the document that any given object will parse.

Care should be taken when choosing attribute names, since use of names defined in the superclass could potentially cause problems when our parser object is used, because a badly chosen name would cause one of our attributes to override an attribute in the superclass and result in our attributes being manipulated for internal parsing purposes by the superclass. We might hope that the SGMLParser class uses attribute names with leading double underscores (__) since this isolates such attributes from access by subclasses such as our own MyParser class.

Remembering Document Details

We now need to define a way of extracting data from the document, but SGMLParser provides a mechanism which notifies us when an interesting part of the document has been read. SGML and HTML are textual formats which are structured by the presence of so-called tags, and in HTML, hyperlinks may be represented in the following way:

The Python Web site

How SGMLParser Operates

An SGMLParser object which is parsing a document recognises starting and ending tags for things such as hyperlinks, and it issues a method call on itself based on the name of the tag found and whether the tag is a starting or ending tag. So, as the above text is recognised by an SGMLParser object (or an object derived from SGMLParser, like MyParser), the following method calls are made internally:

self.start_a(("href", "http://www.python.org"))
self.handle_data("The Python Web site")
self.end_a()

Note that the text between the tags is considered as data, and that the ending tag does not provide any information. The starting tag, however, does provide information in the form of a sequence of attribute names and values, where each name/value pair is placed in a 2-tuple:

# The form of attributes supplied to start tag methods:
# (name, value)
# Examples:
# ("href", "http://www.python.org")
# ("target", "python")

Why SGMLParser Works

Why does SGMLParser issue a method call on itself, effectively telling itself that a tag has been encountered? The basic SGMLParser class surely does not know what to do with such information. Well, if another class inherits from SGMLParser, then such calls are no longer confined to SGMLParser and instead act on methods in the subclass, such as MyParser, where such methods exist. Thus, a customised parser class (eg. MyParser) once instantiated (made into an object) acts like a stack of components, with the lowest level of the stack doing the hard parsing work and passing items of interest to the upper layers - it is a bit like a factory with components being made on the ground floor and inspection of those components taking place in the laboratories in the upper floors!

Class	Activity
...	Listens to reports, records other interesting things
MyParser	Listens to reports, records interesting things
SGMLParser	Parses documents, issuing reports at each step

Introducing Our Customisations

Now, if we want to record the hyperlinks in the document, all we need to do is to define a method called start_a which extracts the hyperlink from the attributes which are provided in the starting a tag. This can be defined as follows:

    # Continuing from above...

    def start_a(self, attributes):
        "Process a hyperlink and its 'attributes'."

        for name, value in attributes:
            if name == "href":
                self.hyperlinks.append(value)

    # More to come...

All we need to do is traverse the attributes list, find appropriately named attributes, and record the value of those attributes.

Retrieving the Details

A nice way of providing access to the retrieved details is to define a method, although Python 2.2 provides additional features to make this more convenient. We shall use the old approach:

    # Continuing from above...

    def get_hyperlinks(self):
        "Return the list of hyperlinks."

        return self.hyperlinks

Trying it Out

Now that we have defined our class, we can instantiate it, making a new MyParser object. After that, it is just a matter of giving it a document to work with:

import urllib, sgmllib

# Get something to work with.
f = urllib.urlopen("http://www.python.org")
s = f.read()

# Try and process the page.
# The class should have been defined first, remember.
myparser = MyParser()
myparser.parse(s)

# Get the hyperlinks.
print myparser.get_hyperlinks()

The print statement should cause a list to be displayed, containing various hyperlinks to locations on the Python home page and other sites.

The Example File

The above example code can be downloaded and executed to see the results.

Finding More Specific Content

Of course, if it is sufficient for you to extract information from a document without worrying about where in the document it came from, then the above level of complexity should suit you perfectly. However, one might want to extract information which only appears in certain places or constructs - a good example of this is the text between starting and ending tags of hyperlinks which we saw above. If we just acquired every piece of text using a handle_data method which recorded everything it saw, then we would not know which piece of text described a hyperlink and which piece of text appeared in any other place in a document.

    # An extension of the above class.
    # This is not very useful.

    def handle_data(self, data):
        "Handle the textual 'data'."

        self.descriptions.append(data)

Here, the descriptions attribute (which we would need to initialise in the __init__ method) would be filled with lots of meaningless textual data. So how can we be more specific? The best approach is to remember not only the content that SGMLParser discovers, but also to remember what kind of content we have seen already.

Remembering Our Position

Let us add some new attributes to the __init__ method.

        # At the end of the __init__ method...

        self.descriptions = []
        self.inside_a_element = 0

The descriptions attribute is defined as we anticipated, but the inside_a_element attribute is used for something different: it will indicate whether or not SGMLParser is currently investigating the contents of an a element - that is, whether SGMLParser is between the starting a tag and the ending a tag.

Let us now add some "logic" to the start_a method, redefining it as follows:

    def start_a(self, attributes):
        "Process a hyperlink and its 'attributes'."

        for name, value in attributes:
            if name == "href":
                self.hyperlinks.append(value)
                self.inside_a_element = 1

Now, we should know when a starting a tag has been seen, but to avoid confusion, we should also change the value of the new attribute when the parser sees an ending a tag. We do this by defining a new method for this case:

    def end_a(self):
        "Record the end of a hyperlink."

        self.inside_a_element = 0

Fortunately, it is not permitted to "nest" hyperlinks, so it is not relevant to wonder what might happen if an ending tag were to be seen after more than one starting tag had been seen in succession.

Recording Relevant Data

Now, given that we can be sure of our position in a document and whether we should record the data that is being presented, we can define the "real" handle_data method as follows:

    def handle_data(self, data):
        "Handle the textual 'data'."

        if self.inside_a_element:
            self.descriptions.append(data)

This method is not perfect, as we shall see, but it does at least avoid recording every last piece of text in the document.

We can now define a method to retrieve the description data:

    def get_descriptions(self):
        "Return a list of descriptions."

        return self.descriptions

And we can add the following line to our test program in order to display the descriptions:

print myparser.get_descriptions()

The Example File

The example code with these modifications can be downloaded and executed to see the results.

Problems with Text

Upon running the modified example, one thing is apparent: there are a few descriptions which do not make sense. Moreover, the number of descriptions does not match the number of hyperlinks. The reason for this is the way that text is found and presented to us by the parser - we may be presented with more than one fragment of text for a particular region of text, so that more than one fragment of text may be signalled between a starting a tag and an ending a tag, even though it is logically one block of text.

We may modify our example by adding another attribute to indicate whether we are just beginning to process a region of text. If this new attribute is set, then we add a description to the list; if not, then we add any text found to the most recent description recorded.

The __init__ method is modified still further:

        # At the end of the __init__ method...

        self.starting_description = 0

Since we can only be sure that a description is being started immediately after a starting a tag has been seen, we redefine the start_a method as follows:

    def start_a(self, attributes):
        "Process a hyperlink and its 'attributes'."

        for name, value in attributes:
            if name == "href":
                self.hyperlinks.append(value)
                self.inside_a_element = 1
                self.starting_description = 1

Now, the handle_data method needs redefining as follows:

    def handle_data(self, data):
        "Handle the textual 'data'."

        if self.inside_a_element:
            if self.starting_description:
                self.descriptions.append(data)
                self.starting_description = 0
            else:
                self.descriptions[-1] += data

Clearly, the method becomes more complicated. We need to detect whether the description is being started and act in the manner discussed above.

The Example File

The example code with these modifications can be downloaded and executed to see the results.

Conclusions

Although the final example file produces some reasonable results - there are some still strange descriptions, however, and we have not taken images used within hyperlinks into consideration - the modifications that were required illustrate that as more attention is paid to the structure of the document, the more effort is required to monitor the origins of information. As a result, we need to maintain state information within the MyParser object in a not-too-elegant way.

For application purposes, the SGMLParser class, its derivatives, and related approaches (such as SAX) are useful for casual access to information, but for certain kinds of querying, they can become more complicated to use than one would initially believe. However, these approaches can be used for another purpose: that of building structures which can be accessed in a more methodical fashion, as we shall see below.

Using XML Parsers

Given a character string s, containing an HTML document which may have been retrieved from a Web service (using an approach described in an earlier section of this document), let us now consider an alternative method of interpreting the contents of this document so that we do not have to manage the complexity of remembering explicitly the structure of the document that we have seen so far. One of the problems with SGMLParser was that access to information in a document happened "serially" - that is, information was presented to us in the order in which it was found - but it may have been more appropriate to access the document information according to the structure of the document, so that we could request all parts of the document corresponding to the hyperlink elements present in that document, before examining each document portion for the text within each hyperlink element.

In the XML world, a standard called the Document Object Model (DOM) has been devised to provide a means of access to document information which permits us to navigate the structure of a document, requesting different sections of that document, and giving us the ability to revisit such sections at any time; the use of Python with XML and the DOM is described in another document. If all Web pages were well-formed XML - that is, they all complied with the expectations and standards set out by the XML specifications - then any XML parser would be sufficient to process any HTML document found on the Web. Unfortunately, many Web pages use less formal variants of HTML which are rejected by XML parsers. Thus, we need to employ particular tools and additional techniques to convert such pages to DOM representations.

Below, we describe how Web pages may be processed using the PyXML toolkit and with the libxml2dom package to obtain a top-level document object. Since both approaches yield an object which is broadly compatible with the DOM standard, the subsequent description of how we then inspect such documents applies regardless of whichever toolkit or package we have chosen.

Using PyXML

It is possible to use Python's XML framework with the kind of HTML found on the Web by employing a special "reader" class which builds a DOM representation from an HTML document, and the consequences of this are described below.

Creating the Reader

An appropriate class for reading HTML documents is found deep in the xml package, and we shall instantiate this class for subsequent use:

from xml.dom.ext.reader import HtmlLib
reader = HtmlLib.Reader()

Of course, there are many different ways of accessing the Reader class concerned, but I have chosen not to import Reader into the common namespace. One good reason for deciding this is that I may wish to import other Reader classes from other packages or modules, and we clearly need a way to distinguish between them. Therefore, I import the HtmlLib name and access the Reader class from within that module.

Loading a Document

Unlike SGMLParser, we do not need to customise any class before we load a document. Therefore, we can "postpone" any consideration of the contents of the document until after the document has been loaded, although it is very likely that you will have some idea of the nature of the contents in advance and will have written classes or functions to work on the DOM representation once it is available. After all, real programs extracting particular information from a certain kind of document do need to know something about the structure of the documents they process, whether that knowledge is put in a subclass of a parser (as in SGMLParser) or whether it is "encoded" in classes and functions which manipulate the DOM representation.

Anyway, let us load the document and obtain a Document object:

doc = reader.fromString(s)

Note that the "top level" of a DOM representation is always a Document node object, and this is what doc refers to immediately after the document is loaded.

Using libxml2dom

Obtaining documents using libxml2dom is slightly more straightforward:

import libxml2dom
doc = libxml2dom.parseString(s, html=1)

If the document text is well-formed XML, we could omit the html parameter or set it to have a false value. However, if we are not sure whether the text is well-formed, no significant issues will arise from setting the parameter in the above fashion.

Deciding What to Extract

Now, it is appropriate to decide which information is to be found and retrieved from the document, and this is where some tasks appear easier than with SGMLParser (and related frameworks). Let us consider the task of extracting all the hyperlinks from the document; we can certainly find all the hyperlink elements as follows:

a_elements = doc.getElementsByTagName("a")

Since hyperlink elements comprise the starting a tag, the ending a tag, and all data between them, the value of the a_elements variable should be a list of objects representing regions in the document which would appear like this:

The Python Web site

Querying Elements

To make the elements easier to deal with, each object in the list is not the textual representation of the element as given above. Instead, an object is created for each element which provides a more convenient level of access to the details. We can therefore obtain a reference to such an object and find out more about the element it represents:

# Get the first element in the list. We don't need to use a separate variable,
# but it makes it clearer.
first = a_elements[0]
# Now display the value of the "href" attribute.
print first.getAttribute("href")

What is happening here is that the first object (being the first a element in the list of those found) is being asked to return the value of the attribute whose name is href, and if such an attribute exists, a string is returned containing the contents of the attribute: in the case of the above example, this would be...

http://www.python.org

If the href attribute had not existed, such as in the following example element, then a value of None would have been returned.

This is not a hyperlink. It is a target.

Namespaces

Previously, this document recommended the usage of namespaces and the getAttributeNS method, rather than the getAttribute method. Whilst XML processing may involve extensive use of namespaces, some HTML parsers do not appear to expose them quite as one would expect: for example, not associating the XHTML namespace with XHTML elements in a document. Thus, it can be advisable to ignore namespaces unless their usage is unavoidable in order to distinguish between elements in mixed-content documents (XHTML combined with SVG, for example).

Finding More Specific Content

We are already being fairly specific, in a sense, in the way that we have chosen to access the a elements within the document, since we start from a particular point in the document's structure and search for elements from there. In the SGMLParser examples, we decided to look for descriptions of hyperlinks in the text which is enclosed between the starting and ending tags associated with hyperlinks, and we were largely successful with that, although there were some issues that could have been handled better. Here, we shall attempt to find everything that is descriptive within hyperlink elements.

Elements, Nodes and Child Nodes

Each hyperlink element is represented by an object whose attributes can be queried, as we did above in order to get the href attribute's value. However, elements can also be queried about their contents, and such contents take the form of objects which represent "nodes" within the document. (The nature of XML documents is described in another introductory document which discusses the DOM.) In this case, it is interesting for us to inspect the nodes which reside within (or under) each hyperlink element, and since these nodes are known generally as "child nodes", we access them through the childNodes attribute on each so-called Node object.

# Get the child nodes of the first "a" element.
nodes = first.childNodes

Node Types

Nodes are the basis of any particular piece of information found in an XML document, so any element found in a document is based on a node and can be explicitly identified as an element by checking its "node type":

print first.nodeType
# A number is returned which corresponds to one of the special values listed in
# the xml.dom.Node class. Since elements inherit from that class, we can access
# these values on 'first' itself!
print first.nodeType == first.ELEMENT_NODE
# If first is an element (it should be) then display the value 1.

One might wonder how this is useful, since the list of hyperlink elements, for example, is clearly a list of elements - that is, after all, what we asked for. However, if we ask an element for a list of "child nodes", we cannot immediately be sure which of these nodes are elements and which are, for example, pieces of textual data. Let us therefore examine the "child nodes" of first to see which of them are textual:

for node in first.childNodes:
    if node.nodeType == node.TEXT_NODE:
        print "Found a text node:", node.nodeValue

Navigating the Document Structure

If we wanted only to get the descriptive text within each hyperlink element, then we would need to visit all nodes within each element (the "child nodes") and record the value of the textual elements. However, this would not quite be enough - consider the following document region:

A really important page.

Within the a element, there are text nodes and an em element - the text within that element is not directly available as a "child node" of the a element. If we did not consider textual child nodes of each child node, then we would miss important information. Consequently, it becomes essential to recursively descend inside the a element collecting child node values. This is not as hard as it sounds, however:

def collect_text(node):
    "A function which collects text inside 'node', returning that text."

    s = ""
    for child_node in node.childNodes:
        if child_node.nodeType == child_node.TEXT_NODE:
            s += child_node.nodeValue
        else:
            s += collect_text(child_node)
    return s

# Call 'collect_text' on 'first', displaying the text found.
print collect_text(first)

To contrast this with the SGMLParser approach, we see that much of the work done in that example to extract textual information is distributed throughout the MyParser class, whereas the above function, which looks quite complicated, gathers the necessary operations into a single place, thus making it look complicated.

Getting Document Regions as Text

Interestingly, it is easier to retrieve whole sections of the original document as text for each of the child nodes, thus collecting the complete contents of the a element as text. For this, we just need to make use of a function provided in the xml.dom.ext package:

from xml.dom.ext import PrettyPrint
# In order to avoid getting the "a" starting and ending tags, prettyprint the
# child nodes.
s = ""
for child_node in a_elements[0]:
    s += PrettyPrint(child_node)
# Display the region of the original document between the tags.
print s

Unfortunately, documents produced by libxml2dom do not work with PrettyPrint. However, we can use a method on each node object instead:

# In order to avoid getting the "a" starting and ending tags, prettyprint the
# child nodes.
s = ""
for child_node in a_elements[0]:
    s += child_node.toString(prettyprint=1)
# Display the region of the original document between the tags.
print s

It is envisaged that libxml2dom will eventually work better with such functions and tools.