Multiprocessing useless with urllib2?

by

Ah here comes yet another discussion about the GIL. Well here’s the thing. Fetching content with urllib2 is going to be mostly IO-bound. Native threading AND multiprocessing will both have the same performance when the task is IO-bound (threading only becomes a problem when it’s CPU-bound). Yes you can speed it up, I’ve done it myself using python threads and something like 10 downloader threads.

Basically you use a producer-consumer model with one thread (or process) producing urls to download, and N threads (or processes) consuming from that queue and making requests to the server.

Here’s some pseudo-code:

# Make sure that the queue is thread-safe!!

def producer(self):
    # Only need one producer, although you could have multiple
    with fh = open('urllist.txt', 'r'):
        for line in fh:
            self.queue.enqueue(line.strip())

def consumer(self):
    # Fire up N of these babies for some speed
    while True:
        url = self.queue.dequeue()
        dh = urllib2.urlopen(url)
        with fh = open('/dev/null', 'w'): # gotta put it somewhere
            fh.write(dh.read())

Now if you’re downloading very large chunks of data (hundreds of MB) and a single request completely saturates the bandwidth, then yes running multiple downloads is pointless. The reason you run multiple downloads (generally) is because requests are small and have a relatively high latency / overhead.

More Related Contents:

  1. multiprocessing: How do I share a dict among multiple processes?
  2. Proxy with urllib2
  3. Is shared readonly data copied to different processes for multiprocessing?
  4. Python Process Pool non-daemonic?
  5. Using an HTTP PROXY – Python [duplicate]
  6. Python: download files from google drive using url
  7. Python handling socket.error: [Errno 104] Connection reset by peer
  8. Script using multiprocessing module does not terminate
  9. What can multiprocessing and dill do together?
  10. Using Python’s Multiprocessing module to execute simultaneous and separate SEAWAT/MODFLOW model runs
  11. python ignore certificate validation urllib2
  12. Python command line input in a process
  13. yet another confusion with multiprocessing error, ‘module’ object has no attribute ‘f’
  14. Multiprocessing Bomb
  15. Simple Python Multiprocessing function in Spyder doesn’t output results
  16. Python multiprocessing.Pool: AttributeError
  17. Python: Getting a traceback from a multiprocessing.Process
  18. Python multiprocessing example not working
  19. Multiprocessing launching too many instances of Python VM
  20. Modify object in python multiprocessing
  21. Python Multiprocess Pool. How to exit the script when one of the worker process determines no more work needs to be done?
  22. open url from pythonanywhere
  23. Submitting to a web form using python
  24. urllib2 HTTP Error 400: Bad Request
  25. python multiprocessing: some functions do not return when they are complete (queue material too big)
  26. Share a list between different processes?
  27. multiprocessing pool example does not work and freeze the kernel
  28. Limiting number of processes in multiprocessing python
  29. Python: Writing to a single file with queue while using multiprocessing Pool
  30. Using python multiprocessing with different random seed for each process

Leave a Comment