Python urllib2.urlopen() is slow, need a better way to read several urls

by

I’m rewriting Dumb Guy’s code below using modern Python modules like threading and Queue.

import threading, urllib2
import Queue

urls_to_load = [
'http://stackoverflow.com/',
'http://slashdot.org/',
'http://www.archive.org/',
'http://www.yahoo.co.jp/',
]

def read_url(url, queue):
    data = urllib2.urlopen(url).read()
    print('Fetched %s from %s' % (len(data), url))
    queue.put(data)

def fetch_parallel():
    result = Queue.Queue()
    threads = [threading.Thread(target=read_url, args = (url,result)) for url in urls_to_load]
    for t in threads:
        t.start()
    for t in threads:
        t.join()
    return result

def fetch_sequencial():
    result = Queue.Queue()
    for url in urls_to_load:
        read_url(url,result)
    return result

Best time for find_sequencial() is 2s. Best time for fetch_parallel() is 0.9s.

Also it is incorrect to say thread is useless in Python because of GIL. This is one of those case when thread is useful in Python because the the threads are blocked on I/O. As you can see in my result the parallel case is 2 times faster.

More Related Contents:

  1. What is the fastest way to send 100,000 HTTP requests in Python?
  2. Using an HTTP PROXY – Python [duplicate]
  3. Python-Requests close http connection
  4. Python: HTTP Post a large file with streaming
  5. Python urllib2 with keep alive
  6. Python urllib2 Progress Hook
  7. What should I do if socket.setdefaulttimeout() is not working?
  8. Unescape Python Strings From HTTP
  9. How do I catch a specific HTTP error in Python?
  10. Ideal method for sending multiple HTTP requests over Python? [duplicate]
  11. Import error: No module name urllib2
  12. Python Requests library redirect new url
  13. Python: download files from google drive using url
  14. urllib2 read to Unicode
  15. python ignore certificate validation urllib2
  16. Handling urllib2’s timeout? – Python
  17. Fastest parallel requests in Python
  18. Can an asyncio event loop run in the background without suspending the Python interpreter?
  19. How to get response SSL certificate from requests in python?
  20. Read timeout using either urllib2 or any other http library
  21. How to send zip files in the python Flask framework?
  22. urllib2 HTTP Error 400: Bad Request
  23. Limiting number of processes in multiprocessing python
  24. Checking if a website is up via Python
  25. How to get the raw content of a response in requests with Python?
  26. Why do I get “‘str’ object has no attribute ‘read'” when trying to use `json.load` on a string? [duplicate]
  27. Multiprocessing useless with urllib2?
  28. How do I use threading in Python?
  29. How to resolve URLError:
  30. Python requests.exception.ConnectionError: connection aborted “BadStatusLine”

Leave a Comment