Try emulating your browser with selenium. This script will pop up the save as dialog for the webpage. You will still have to figure out how to emulate pressing enter for download to start as the file dialog is out of selenium’s reach (how you do it is also OS dependent). from selenium import webdriver … Read more
If you want to pass q as a parameter in the URL using requests, use the params argument, not data (see Passing Parameters In URLs): r = requests.get(‘http://stackoverflow.com’, params=data) This will request https://stackoverflow.com/?q=%5Bpython%5D , which isn’t what you are looking for. You really want to POST to a form. Try this: r = requests.post(‘https://stackoverflow.com/search’, data=data) … Read more
This is how you do it. from urllib import request, parse data = parse.urlencode(<your data dict>).encode() req = request.Request(<your url>, data=data) # this will make the method “POST” resp = request.urlopen(req)
From the docs Note that params output from urlencode is encoded to bytes before it is sent to urlopen as data: data = urllib.parse.urlencode(d).encode(“utf-8”) req = urllib.request.Request(url) with urllib.request.urlopen(req,data=data) as f: resp = f.read() print(resp)
Python 3 from urllib.error import HTTPError Python 2 from urllib2 import HTTPError Just catch HTTPError, handle it, and if it’s not Error 404, simply use raise to re-raise the exception. See the Python tutorial. Here is a complete example for Python 2: import urllib2 from urllib2 import HTTPError try: urllib2.urlopen(“some url”) except HTTPError as err: … Read more
You’ll have to add the key separately, after the creating the initial dict: params = {‘apple’: apple} if orange is not None: params[‘orange’] = orange params = urllib.urlencode(params) Python has no syntax to define a key as conditional; you could use a dict comprehension if you already had everything in a sequence: params = urllib.urlencode({k: … Read more
From the Python docs: import urllib.request req = urllib.request.Request( url, data=None, headers={ ‘User-Agent’: ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36’ } ) f = urllib.request.urlopen(req) print(f.read().decode(‘utf-8’))
Pandas can do this right out of the box, saving you from having to parse the html yourself. to_html() extracts all tables from your html and puts them in a list of dataframes. to_csv() can be used to convert each dataframe to a csv file. For the web page in your example, the relevant table … Read more
When you download a file with urllib or urllib2, you can find out whether a charset header was transmitted: fp = urllib2.urlopen(request) charset = fp.headers.getparam(‘charset’) You can use BeautifulSoup to locate a meta element in the HTML: soup = BeatifulSoup.BeautifulSoup(data) meta = soup.findAll(‘meta’, {‘http-equiv’:lambda v:v.lower()==’content-type’}) If neither is available, browsers typically fall back to user … Read more
I have reproduced what you are seeing: import urllib, os link = “http://python.org” print “opening url:”, link site = urllib.urlopen(link) meta = site.info() print “Content-Length:”, meta.getheaders(“Content-Length”)[0] f = open(“out.txt”, “r”) print “File on disk:”,len(f.read()) f.close() f = open(“out.txt”, “w”) f.write(site.read()) site.close() f.close() f = open(“out.txt”, “r”) print “File on disk after download:”,len(f.read()) f.close() print “os.stat().st_size … Read more