You need to seek to the beginning of compressedFile after writing to it but before passing it to gzip.GzipFile(). Otherwise it will be read from the end by gzip module and will appear as an empty file to it. See below: #! /usr/bin/env python import urllib2 import StringIO import gzip baseURL = “https://www.kernel.org/pub/linux/docs/man-pages/” filename = … Read more
The simplest way to limit number of concurrent connections is to use a thread pool: #!/usr/bin/env python from itertools import izip, repeat from multiprocessing.dummy import Pool # use threads for I/O bound tasks from urllib2 import urlopen def fetch(url_data): try: return url_data[0], urlopen(*url_data).read(), None except EnvironmentError as e: return url_data[0], None, str(e) if __name__==”__main__”: pool … Read more
The reason that “the dog” returns a 400 Error is because you aren’t escaping the string for a URL. If you do this: import urllib, urllib2 quoted_query = urllib.quote(query) host=”http://www.bing.com/search?q=%s&go=&qs=n&sk=&sc=8-13&first=%s” % (quoted_query, page) req = urllib2.Request(host) req.add_header(‘User-Agent’, User_Agent) response = urllib2.urlopen(req) It will work. However I highly suggest you use requests instead of using urllib/urllib2/httplib. … Read more
If you want to pass q as a parameter in the URL using requests, use the params argument, not data (see Passing Parameters In URLs): r = requests.get(‘http://stackoverflow.com’, params=data) This will request https://stackoverflow.com/?q=%5Bpython%5D , which isn’t what you are looking for. You really want to POST to a form. Try this: r = requests.post(‘https://stackoverflow.com/search’, data=data) … Read more
You can easily get D by just asking for the current URL. req = urllib2.Request(starturl, datagen, headers) res = urllib2.urlopen(req) finalurl = res.geturl() To deal with the intermediate redirects you’ll probably need to build your own opener, using HTTPRedirectHandler that records the redirects.
Basically what you do is correct. Looking at redmine docs you linked to, it seems that suffix after the dot in the url denotes type of posted data (.json for JSON, .xml for XML), which agrees with the response you get – Processing by AttachmentsController#upload as XML. I guess maybe there’s a bug in docs … Read more
Free accounts on PythonAnywhere are restricted to a whitelist of sites, http/https only, and access goes via a proxy. There’s more info here: PythonAnywhere wiki: “why do I get a 403 forbidden error when opening a url?”
From the docs Note that params output from urlencode is encoded to bytes before it is sent to urlopen as data: data = urllib.parse.urlencode(d).encode(“utf-8”) req = urllib.request.Request(url) with urllib.request.urlopen(req,data=data) as f: resp = f.read() print(resp)
Python 3 from urllib.error import HTTPError Python 2 from urllib2 import HTTPError Just catch HTTPError, handle it, and if it’s not Error 404, simply use raise to re-raise the exception. See the Python tutorial. Here is a complete example for Python 2: import urllib2 from urllib2 import HTTPError try: urllib2.urlopen(“some url”) except HTTPError as err: … Read more
If the URL is returning valid JSON-encoded data, use the json library to decode that: import urllib2 import json response = urllib2.urlopen(‘https://api.instagram.com/v1/tags/pizza/media/XXXXXX’) data = json.load(response) print data