What is the fastest way to send 100,000 HTTP requests in Python?

by

Twistedless solution:

from urlparse import urlparse
from threading import Thread
import httplib, sys
from Queue import Queue

concurrent = 200

def doWork():
    while True:
        url = q.get()
        status, url = getStatus(url)
        doSomethingWithResult(status, url)
        q.task_done()

def getStatus(ourl):
    try:
        url = urlparse(ourl)
        conn = httplib.HTTPConnection(url.netloc)   
        conn.request("HEAD", url.path)
        res = conn.getresponse()
        return res.status, ourl
    except:
        return "error", ourl

def doSomethingWithResult(status, url):
    print status, url

q = Queue(concurrent * 2)
for i in range(concurrent):
    t = Thread(target=doWork)
    t.daemon = True
    t.start()
try:
    for url in open('urllist.txt'):
        q.put(url.strip())
    q.join()
except KeyboardInterrupt:
    sys.exit(1)

This one is slighty faster than the twisted solution and uses less CPU.

More Related Contents:

  1. Python urllib2.urlopen() is slow, need a better way to read several urls
  2. Ideal method for sending multiple HTTP requests over Python? [duplicate]
  3. How to download a file over HTTP?
  4. multiprocessing.Pool: When to use apply, apply_async or map?
  5. Python requests – print entire http request (raw)?
  6. Python Requests library redirect new url
  7. How to limit concurrency with Python asyncio?
  8. Problem HTTP error 403 in Python 3 Web Scraping
  9. Python urllib2, basic HTTP authentication, and tr.im
  10. Python: HTTP Post a large file with streaming
  11. Solving embarassingly parallel problems using Python multiprocessing
  12. Fastest parallel requests in Python
  13. Simple URL GET/POST function in Python
  14. Only download a part of the document using python requests
  15. Can an asyncio event loop run in the background without suspending the Python interpreter?
  16. How to get http headers in flask?
  17. How to stop BaseHTTPServer.serve_forever() in a BaseHTTPRequestHandler subclass?
  18. What should I do if socket.setdefaulttimeout() is not working?
  19. Read timeout using either urllib2 or any other http library
  20. Save a large file using the Python requests library [duplicate]
  21. Python requests speed up using keep-alive
  22. Flask – POST Error 405 Method Not Allowed
  23. How do I catch a specific HTTP error in Python?
  24. How to send zip files in the python Flask framework?
  25. Checking if a website is up via Python
  26. How to get the raw content of a response in requests with Python?
  27. Is there a library for retrieving a file from a remote zip? [closed]
  28. Python seek on remote file using HTTP
  29. How do I use threading in Python?
  30. Python requests.exception.ConnectionError: connection aborted “BadStatusLine”

Leave a Comment