Python - urllib - Revision history

PeterHarding at 06:48, 11 March 2017

2017-03-11T06:48:02Z

← Older revision		Revision as of 16:48, 11 March 2017
Line 1:		Line 1:
	~~= References =~~

	* http://www.voidspace.org.uk/python/articles/urllib2.shtml		Article from - http://www.voidspace.org.uk/python/articles/urllib2.shtml


	= URLLIB Tutorial: urllib2 - The Missing Manual =		= URLLIB Tutorial: urllib2 - The Missing Manual =


	== HOWTO Fetch Internet Resources with Python ==		== HOWTO Fetch Internet Resources with Python ==

PeterHarding at 04:04, 1 August 2015

2015-08-01T04:04:17Z

← Older revision		Revision as of 14:04, 1 August 2015
Line 439:		Line 439:

	[[Category:Python]]		[[Category:Python]]
	[[Category:. httplib]]		[[Category:Python httplib]]
	[[Category:. urllib]]		[[Category:Python urllib]]
	[[Category:Internet]]		[[Category:Internet]]

PeterHarding at 01:29, 2 April 2009

2009-04-02T01:29:47Z

← Older revision		Revision as of 11:29, 2 April 2009
Line 77:		Line 77:
	== Data ==		== Data ==

	Sometimes you want to send data to a URL (often the URL will refer to a CGI (Common Gateway Interface) script ~~[http://esbinfo:8090/pages/editpage.action#id8 ][1]~~ or other web application). With HTTP, this is often done using what's known as a <b>POST</b> request. This is often what your browser does when you submit a HTML form that you filled in on the web. Not all POSTs have to come from forms: you can use a POST to transmit arbitrary data to your own application. In the common case of HTML forms, the data needs to be encoded in a standard way, and then passed to the Request object as the <tt class="docutils literal"><span class="pre">data</span></tt> argument. The encoding is done using a function from the <tt class="docutils literal"><span class="pre">urllib</span></tt> library <i>not</i> from <tt class="docutils literal"><span class="pre">urllib2</span></tt>.		Sometimes you want to send data to a URL (often the URL will refer to a CGI (Common Gateway Interface) script or other web application). With HTTP, this is often done using what's known as a <b>POST</b> request. This is often what your browser does when you submit a HTML form that you filled in on the web. Not all POSTs have to come from forms: you can use a POST to transmit arbitrary data to your own application. In the common case of HTML forms, the data needs to be encoded in a standard way, and then passed to the Request object as the <tt class="docutils literal"><span class="pre">data</span></tt> argument. The encoding is done using a function from the <tt class="docutils literal"><span class="pre">urllib</span></tt> library <i>not</i> from <tt class="docutils literal"><span class="pre">urllib2</span></tt>.

	<pre>		<pre>
Line 111:		Line 111:
	We'll discuss here one particular HTTP header, to illustrate how to add headers to your HTTP request.		We'll discuss here one particular HTTP header, to illustrate how to add headers to your HTTP request.

	Some websites ~~[http://esbinfo:8090/pages/editpage.action#id9 ][2]~~ dislike being browsed by programs, or send different versions to different browsers ~~[http://esbinfo:8090/pages/editpage.action#id10 ][3]~~ . By default urllib2 identifies itself as <tt class="docutils literal"><span class="pre">Python-urllib/x.y</span></tt> (where <tt class="docutils literal"><span class="pre">x</span></tt> and <tt class="docutils literal"><span class="pre">y</span></tt> are the major and minor version numbers of the Python release, e.g. <tt class="docutils literal"><span class="pre">Python-urllib/2.5</span></tt>), which may confuse the site, or just plain not work. The way a browser identifies itself is through the <tt class="docutils literal"><span class="pre">User-Agent</span></tt> header ~~[http://esbinfo:8090/pages/editpage.action#id11 ][4]~~. When you create a Request object you can pass a dictionary of headers in. The following example makes the same request as above, but identifies itself as a version of Internet Explorer ~~[http://esbinfo:8090/pages/editpage.action#id12 ][5]~~.		Some websites dislike being browsed by programs, or send different versions to different browsers. By default urllib2 identifies itself as <tt class="docutils literal"><span class="pre">Python-urllib/x.y</span></tt> (where <tt class="docutils literal"><span class="pre">x</span></tt> and <tt class="docutils literal"><span class="pre">y</span></tt> are the major and minor version numbers of the Python release, e.g. <tt class="docutils literal"><span class="pre">Python-urllib/2.5</span></tt>), which may confuse the site, or just plain not work. The way a browser identifies itself is through the <tt class="docutils literal"><span class="pre">User-Agent</span></tt> header. When you create a Request object you can pass a dictionary of headers in. The following example makes the same request as above, but identifies itself as a version of Internet Explorer.

	<pre>		<pre>
Line 131:		Line 131:


	The response also has two useful methods. See the section on ~~[http://esbinfo:8090/pages/editpage.action#info-and-geturl info and geturl]~~ which comes after we have a look at what happens when things go wrong.		The response also has two useful methods. See the section on which comes after we have a look at what happens when things go wrong.

	</div></div><div class="section">		</div></div><div class="section">
Line 387:		Line 387:
	= Proxies =		= Proxies =

	<b>urllib2</b> will auto-detect your proxy settings and use those. This is through the <tt class="docutils literal"><span class="pre">ProxyHandler</span></tt> which is part of the normal handler chain. Normally that's a good thing, but there are occasions when it may not be helpful ~~[http://esbinfo:8090/pages/editpage.action#id13 ][6]~~. One way to do this is to setup our own <tt class="docutils literal"><span class="pre">ProxyHandler</span></tt>, with no proxies defined. This is done using similar steps to setting up a [http://www.voidspace.org.uk/python/articles/authentication.shtml Basic Authentication] handler :		<b>urllib2</b> will auto-detect your proxy settings and use those. This is through the <tt class="docutils literal"><span class="pre">ProxyHandler</span></tt> which is part of the normal handler chain. Normally that's a good thing, but there are occasions when it may not be helpful. One way to do this is to setup our own <tt class="docutils literal"><span class="pre">ProxyHandler</span></tt>, with no proxies defined. This is done using similar steps to setting up a [http://www.voidspace.org.uk/python/articles/authentication.shtml Basic Authentication] handler :

	>>> proxy_support = urllib2.ProxyHandler({})<br />>>> opener = urllib2.build_opener(proxy_support)<br />>>> urllib2.install_opener(opener)<br />		>>> proxy_support = urllib2.ProxyHandler({})<br />>>> opener = urllib2.build_opener(proxy_support)<br />>>> urllib2.install_opener(opener)<br />

PeterHarding: /* Fetching URLs */

2008-11-12T06:27:34Z

Fetching URLs

← Older revision		Revision as of 16:27, 12 November 2008
Line 67:		Line 67:
	Note that urllib2 makes use of the same Request interface to handle all URL schemes. For example, you can make an FTP request like so :		Note that urllib2 makes use of the same Request interface to handle all URL schemes. For example, you can make an FTP request like so :

	<~~div class="pysrc"><span class="pytext"~~>req~~</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class~~=~~"pytext">~~urllib2~~</span><span class="pyoperator">~~.~~</span><span class="pytext">~~Request~~</span><span class="pyoperator">~~(~~</span><span class="pystring">~~'ftp://example.com/'~~</span><span class="pyoperator">~~)</~~span><span class="pytext"></span></div~~>		<pre>
			req = urllib2.Request('ftp://example.com/')
			</pre>

	In the case of HTTP, there are two extra things that Request objects allow you to do: First, you can pass data to be sent to the server. Second, you can pass extra information ("metadata") <i>about</i> the data or the about request itself, to the server - this information is sent as HTTP "headers". Let's look at each of these in turn.		In the case of HTTP, there are two extra things that Request objects allow you to do: First, you can pass data to be sent to the server. Second, you can pass extra information ("metadata") <i>about</i> the data or the about request itself, to the server - this information is sent as HTTP "headers". Let's look at each of these in turn.

PeterHarding: /* Fetching URLs */

2008-11-12T06:26:21Z

Fetching URLs

← Older revision		Revision as of 16:26, 12 November 2008
Line 42:		Line 42:
	= Fetching URLs =		= Fetching URLs =

	The simplest way to use urllib2 is as follows :		The simplest way to use urllib2 is as follows:

	<~~div class="pysrc"><span class="pykeyword"~~>import~~</span> <span class="pytext">~~urllib2~~</span><br /><span class="pytext">~~response~~</span> <span class="pyoperator"><nowiki>~~=~~</nowiki></span> <span class="pytext">~~urllib2~~</span><span class="pyoperator">~~.~~</span><span class="pytext">~~urlopen~~</span><span class="pyoperator">~~(~~</span><span class="pystring">~~'http://python.org/'~~</span><span class="pyoperator">~~)~~</span><br /><span class="pytext">~~html~~</span> <span class~~=~~"pyoperator"><nowiki>=</nowiki></span> <span class="pytext">~~response~~</span><span class="pyoperator">~~.~~</span><span class="pytext">~~read~~</span><span class="pyoperator">~~(~~</span><span class="pyoperator">~~)</~~span><span class="pytext"></span></div~~>		<pre>
			import urllib2
			response = urllib2.urlopen('http://python.org/')
			html = response.read()
			</pre>

	Many uses of urllib2 will be that simple (note that instead of an 'http:' URL we could have used an URL starting with 'ftp:', 'file:', etc.). However, it's the purpose of this tutorial to explain the more complicated cases, concentrating on HTTP.		Many uses of urllib2 will be that simple (note that instead of an 'http:' URL we could have used an URL starting with 'ftp:', 'file:', etc.). However, it's the purpose of this tutorial to explain the more complicated cases, concentrating on HTTP.

PeterHarding: /* Fetching URLs */

2008-11-12T06:25:52Z

Fetching URLs

← Older revision		Revision as of 16:25, 12 November 2008
Line 50:		Line 50:
	HTTP is based on requests and responses - the client makes requests and servers send responses. urllib2 mirrors this with a <tt class="docutils literal"><span class="pre">Request</span></tt> object which represents the HTTP request you are making. In its simplest form you create a Request object that specifies the URL you want to fetch. Calling <tt class="docutils literal"><span class="pre">urlopen</span></tt> with this Request object returns a response object for the URL requested. This response is a file-like object, which means you can for example call .read() on the response :		HTTP is based on requests and responses - the client makes requests and servers send responses. urllib2 mirrors this with a <tt class="docutils literal"><span class="pre">Request</span></tt> object which represents the HTTP request you are making. In its simplest form you create a Request object that specifies the URL you want to fetch. Calling <tt class="docutils literal"><span class="pre">urlopen</span></tt> with this Request object returns a response object for the URL requested. This response is a file-like object, which means you can for example call .read() on the response :

	<~~div class="pysrc"><span class="pykeyword"~~>import~~</span> <span class="pytext">~~urllib2~~</span><br /><br /><span class="pytext">~~req~~</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class~~=~~"pytext">~~urllib2~~</span><span class="pyoperator">~~.~~</span><span class="pytext">~~Request~~</span><span class="pyoperator">~~(~~</span><span class="pystring">~~'http://www.voidspace.org.uk'~~</span><span class="pyoperator">~~)~~</span><br /><span class="pytext">~~response~~</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class~~=~~"pytext">~~urllib2~~</span><span class="pyoperator">~~.~~</span><span class="pytext">~~urlopen~~</span><span class="pyoperator">~~(~~</span><span class="pytext">~~req~~</span><span class="pyoperator">~~)~~</span><br /><span class="pytext">~~the_page~~</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class~~=~~"pytext">~~response~~</span><span class="pyoperator">~~.~~</span><span class="pytext">~~read~~</span><span class~~=~~"pyoperator">~~(</~~span><span class="pyoperator">)<~~/~~span><span class="pytext"><~~/~~span>~~</~~div~~>		<pre>
			import urllib2

			req = urllib2.Request('http://www.voidspace.org.uk')
			response = urllib2.urlopen(req)
			the_page = response.read()

			Note that urllib2 makes use of the same Request interface to handle all URL schemes. For example, you can make an FTP request like so :
			req = urllib2.Request('ftp://example.com/')
			</pre>

	Note that urllib2 makes use of the same Request interface to handle all URL schemes. For example, you can make an FTP request like so :		Note that urllib2 makes use of the same Request interface to handle all URL schemes. For example, you can make an FTP request like so :

PeterHarding: /* Data */

2008-11-12T06:25:28Z

Data

@@ Line 64: / Line 64: @@
 Sometimes you want to send data to a URL (often the URL will refer to a CGI (Common Gateway Interface) script [http://esbinfo:8090/pages/editpage.action#id8 ][1] or other web application). With HTTP, this is often done using what's known as a <b>POST</b> request. This is often what your browser does when you submit a HTML form that you filled in on the web. Not all POSTs have to come from forms: you can use a POST to transmit arbitrary data to your own application. In the common case of HTML forms, the data needs to be encoded in a standard way, and then passed to the Request object as the <tt class="docutils literal"><span class="pre">data</span></tt> argument. The encoding is done using a function from the <tt class="docutils literal"><span class="pre">urllib</span></tt> library <i>not</i> from <tt class="docutils literal"><span class="pre">urllib2</span></tt>.
-<div class="pysrc"><span class="pykeyword">import</span> <span class="pytext">urllib</span><br /><span class="pykeyword">import</span> <span class="pytext">urllib2</span><br /><br /><span class="pytext">url</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class="pystring">'http://www.someserver.com/cgi-bin/register.cgi'</span><br /><span class="pytext">values</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class="pyoperator">{</span><span class="pystring">'name'</span> <span class="pyoperator"><nowiki>:</nowiki></span> <span class="pystring">'Michael Foord'</span><span class="pyoperator">,</span><br /><span class="pystring">'location'</span> <span class="pyoperator"><nowiki>:</nowiki></span> <span class="pystring">'Northampton'</span><span class="pyoperator">,</span><br /><span class="pystring">'language'</span> <span class="pyoperator"><nowiki>:</nowiki></span> <span class="pystring">'Python'</span> <span class="pyoperator">}</span><br /><br /><span class="pytext">data</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class="pytext">urllib</span><span class="pyoperator">.</span><span class="pytext">urlencode</span><span class="pyoperator">(</span><span class="pytext">values</span><span class="pyoperator">)</span><br /><span class="pytext">req</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class="pytext">urllib2</span><span class="pyoperator">.</span><span class="pytext">Request</span><span class="pyoperator">(</span><span class="pytext">url</span><span class="pyoperator">,</span> <span class="pytext">data</span><span class="pyoperator">)</span><br /><span class="pytext">response</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class="pytext">urllib2</span><span class="pyoperator">.</span><span class="pytext">urlopen</span><span class="pyoperator">(</span><span class="pytext">req</span><span class="pyoperator">)</span><br /><span class="pytext">the_page</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class="pytext">response</span><span class="pyoperator">.</span><span class="pytext">read</span><span class="pyoperator">(</span><span class="pyoperator">)</span><span class="pytext"></span></div>
+<pre>
 Note that other encodings are sometimes required (e.g. for file upload from HTML forms - see [http://www.w3.org/TR/REC-html40/interact/forms.html#h-17.13 HTML Specification, Form Submission] for more details).

PeterHarding: /* Headers */

2008-11-12T06:24:54Z

Headers

@@ Line 84: / Line 84: @@
 Some websites [http://esbinfo:8090/pages/editpage.action#id9 ][2] dislike being browsed by programs, or send different versions to different browsers [http://esbinfo:8090/pages/editpage.action#id10 ][3] . By default urllib2 identifies itself as <tt class="docutils literal"><span class="pre">Python-urllib/x.y</span></tt> (where <tt class="docutils literal"><span class="pre">x</span></tt> and <tt class="docutils literal"><span class="pre">y</span></tt> are the major and minor version numbers of the Python release, e.g. <tt class="docutils literal"><span class="pre">Python-urllib/2.5</span></tt>), which may confuse the site, or just plain not work. The way a browser identifies itself is through the <tt class="docutils literal"><span class="pre">User-Agent</span></tt> header [http://esbinfo:8090/pages/editpage.action#id11 ][4]. When you create a Request object you can pass a dictionary of headers in. The following example makes the same request as above, but identifies itself as a version of Internet Explorer [http://esbinfo:8090/pages/editpage.action#id12 ][5].
-<div class="pysrc"><span class="pykeyword">import</span> <span class="pytext">urllib</span><br /><span class="pykeyword">import</span> <span class="pytext">urllib2</span><br /><br /><span class="pytext">url</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class="pystring">'http://www.someserver.com/cgi-bin/register.cgi'</span><br /><span class="pytext">user_agent</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class="pystring">'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'</span><br /><span class="pytext">values</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class="pyoperator">{</span><span class="pystring">'name'</span> <span class="pyoperator"><nowiki>:</nowiki></span> <span class="pystring">'Michael Foord'</span><span class="pyoperator">,</span><br /><span class="pystring">'location'</span> <span class="pyoperator"><nowiki>:</nowiki></span> <span class="pystring">'Northampton'</span><span class="pyoperator">,</span><br /><span class="pystring">'language'</span> <span class="pyoperator"><nowiki>:</nowiki></span> <span class="pystring">'Python'</span> <span class="pyoperator">}</span><br /><span class="pytext">headers</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class="pyoperator">{</span> <span class="pystring">'User-Agent'</span> <span class="pyoperator"><nowiki>:</nowiki></span> <span class="pytext">user_agent</span> <span class="pyoperator">}</span><br /><br /><span class="pytext">data</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class="pytext">urllib</span><span class="pyoperator">.</span><span class="pytext">urlencode</span><span class="pyoperator">(</span><span class="pytext">values</span><span class="pyoperator">)</span><br /><span class="pytext">req</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class="pytext">urllib2</span><span class="pyoperator">.</span><span class="pytext">Request</span><span class="pyoperator">(</span><span class="pytext">url</span><span class="pyoperator">,</span> <span class="pytext">data</span><span class="pyoperator">,</span> <span class="pytext">headers</span><span class="pyoperator">)</span><br /><span class="pytext">response</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class="pytext">urllib2</span><span class="pyoperator">.</span><span class="pytext">urlopen</span><span class="pyoperator">(</span><span class="pytext">req</span><span class="pyoperator">)</span><br /><span class="pytext">the_page</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class="pytext">response</span><span class="pyoperator">.</span><span class="pytext">read</span><span class="pyoperator">(</span><span class="pyoperator">)</span><span class="pytext"></span></div>
+<pre>
 The response also has two useful methods. See the section on [http://esbinfo:8090/pages/editpage.action#info-and-geturl info and geturl] which comes after we have a look at what happens when things go wrong.

PeterHarding: /* Error Codes */

2008-11-12T06:24:26Z

Error Codes

Show changes

PeterHarding: /* Number 1 */

2008-11-12T06:22:33Z

Number 1

@@ Line 141: / Line 141: @@
 === Number 1 ===
-<div class="pysrc"><span class="pykeyword">from</span> <span class="pytext">urllib2</span> <span class="pykeyword">import</span> <span class="pytext">Request</span><span class="pyoperator">,</span> <span class="pytext">urlopen</span><span class="pyoperator">,</span> <span class="pytext">URLError</span><span class="pyoperator">,</span> <span class="pytext">HTTPError</span><br /><span class="pytext">req</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class="pytext">Request</span><span class="pyoperator">(</span><span class="pytext">someurl</span><span class="pyoperator">)</span><br /><span class="pykeyword">try</span><span class="pyoperator"><nowiki>:</nowiki></span><br /><span class="pytext">response</span> <span class="pyoperator"><nowiki>=</nowiki></span> <span class="pytext">urlopen</span><span class="pyoperator">(</span><span class="pytext">req</span><span class="pyoperator">)</span><br /><span class="pykeyword">except</span> <span class="pytext">HTTPError</span><span class="pyoperator">,</span> <span class="pytext">e</span><span class="pyoperator"><nowiki>:</nowiki></span><br /><span class="pykeyword">print</span> <span class="pystring">'The server couldn't fulfill the request.'</span><br /><span class="pykeyword">print</span> <span class="pystring">'Error code: '</span><span class="pyoperator">,</span> <span class="pytext">e</span><span class="pyoperator">.</span><span class="pytext">code</span><br /><span class="pykeyword">except</span> <span class="pytext">URLError</span><span class="pyoperator">,</span> <span class="pytext">e</span><span class="pyoperator"><nowiki>:</nowiki></span><br /><span class="pykeyword">print</span> <span class="pystring">'We failed to reach a server.'</span><br /><span class="pykeyword">print</span> <span class="pystring">'Reason: '</span><span class="pyoperator">,</span> <span class="pytext">e</span><span class="pyoperator">.</span><span class="pytext">reason</span><br /><span class="pykeyword">else</span><span class="pyoperator"><nowiki>:</nowiki></span><br /><span class="pycomment"><nowiki># everything is fine</nowiki></span><span class="pytext"></span></div><div class="note">
+<pre>
 ====Note====