Ideal method for sending multiple HTTP requests over Python? [duplicate]

by

If a lot is really a lot than you probably want use asynchronous io not threads.

requests + gevent = grequests

GRequests allows you to use Requests with Gevent to make asynchronous HTTP Requests easily.

import grequests

urls = [
    'http://www.heroku.com',
    'http://tablib.org',
    'http://httpbin.org',
    'http://python-requests.org',
    'http://kennethreitz.com'
]

rs = (grequests.get(u) for u in urls)
grequests.map(rs)

More Related Contents:

  1. What is the fastest way to send 100,000 HTTP requests in Python?
  2. Python urllib2.urlopen() is slow, need a better way to read several urls
  3. Checking if a website is up via Python
  4. How to limit download rate of HTTP requests in requests python library?
  5. How can I use threading in Python?
  6. urllib2.HTTPError: HTTP Error 403: Forbidden
  7. How do you send a HEAD HTTP request in Python 2?
  8. How to avoid HTTP error 429 (Too Many Requests) python
  9. A good way to get the charset/encoding of an HTTP response in Python
  10. Is there an easy way to request a URL in python and NOT follow redirects?
  11. Python urllib2 with keep alive
  12. Python ‘requests’ library – define specific DNS?
  13. Python Requests package: Handling xml response
  14. How to encode UTF8 filename for HTTP headers? (Python, Django)
  15. Python urllib2 Progress Hook
  16. In Python, how do I decode GZIP encoding?
  17. Why am I getting this error in python ? (httplib)
  18. How do I send a POST request as a JSON?
  19. Using Flask, how do I modify the Cache-Control header for ALL output?
  20. python how to decode http response
  21. How to implement retry mechanism into python requests library?
  22. Python: How do I get key/value pairs from the BaseHTTPRequestHandler HTTP POST handler?
  23. How to run multiple functions at the same time?
  24. Download file using partial download (HTTP)
  25. Python – make a POST request using Python 3 urllib
  26. How to send and receive HTTP POST requests in Python [closed]
  27. Is there a library for retrieving a file from a remote zip? [closed]
  28. Python seek on remote file using HTTP
  29. How do I use threading in Python?
  30. Python requests.exception.ConnectionError: connection aborted “BadStatusLine”

Leave a Comment