python-requests

XHR request URL says does not exist when attempting to parse it’s content

You have several problems: the url should be http://www.whoscored.com/stageplayerstatfeed wrong GET parameters missing important required headers you need response.json(), not response.body The fixed version: import requests url=”http://www.whoscored.com/stageplayerstatfeed” params = { ‘field’: ‘1’, ‘isAscending’: ‘false’, ‘orderBy’: ‘Rating’, ‘playerId’: ‘-1’, ‘stageId’: ‘9155’, ‘teamId’: ’32’ } headers = {‘User-Agent’: ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, … Read more

Requests with multiple connections

You can use HTTP Range header to fetch just part of file (already covered for python here). Just start several threads and fetch different range with each and you’re done 😉 def download(url,start): req = urllib2.Request(‘http://www.python.org/’) req.headers[‘Range’] = ‘bytes=%s-%s’ % (start, start+chunk_size) f = urllib2.urlopen(req) parts[start] = f.read() threads = [] parts = {} # … Read more

Python 3: using requests does not get the full content of a web page

The page is rendered with JavaScript making more requests to fetch additional data. You can fetch the complete page with selenium. from bs4 import BeautifulSoup from selenium import webdriver driver = webdriver.Chrome() url = “https://shop.nordstrom.com/c/womens-dresses-shop?origin=topnav&cm_sp=Top%20Navigation-_-Women-_-Dresses&offset=11&page=3&top=72” driver.get(url) soup = BeautifulSoup(driver.page_source, ‘html.parser’) driver.quit() print(soup.prettify()) For other solutions see my answer to Scraping Google Finance (BeautifulSoup)

Progress of Python requests post

requests doesn’t support upload streaming e.g.: import os import sys import requests # pip install requests class upload_in_chunks(object): def __init__(self, filename, chunksize=1 << 13): self.filename = filename self.chunksize = chunksize self.totalsize = os.path.getsize(filename) self.readsofar = 0 def __iter__(self): with open(self.filename, ‘rb’) as file: while True: data = file.read(self.chunksize) if not data: sys.stderr.write(“\n”) break self.readsofar += … Read more

Python Requests – SSL error for client side cert

I had this same problem. The verify parameter refers to the server’s certificate. You want the cert parameter to specify your client certificate. import requests cert_file_path = “cert.pem” key_file_path = “key.pem” url = “https://example.com/resource” params = {“param_1”: “value_1”, “param_2”: “value_2”} cert = (cert_file_path, key_file_path) r = requests.get(url, params=params, cert=cert)

how to post multiple value with same key in python requests?

Dictionary keys must be unique, you can’t repeat them. You’d use a sequence of key-value tuples instead, and pass this to data: requests.post(url, data=[(‘interests’, ‘football’), (‘interests’, ‘basketball’)]) Alternatively, make the values of the data dictionary lists; each value in the list is used as a separate parameter entry: requests.post(url, data={‘interests’: [‘football’, ‘basketball’]}) Demo POST to … Read more