Limiting number of processes in multiprocessing python

by

The simplest way to limit number of concurrent connections is to use a thread pool:

#!/usr/bin/env python
from itertools import izip, repeat
from multiprocessing.dummy import Pool # use threads for I/O bound tasks
from urllib2 import urlopen

def fetch(url_data):
    try:
        return url_data[0], urlopen(*url_data).read(), None
    except EnvironmentError as e:
        return url_data[0], None, str(e)

if __name__=="__main__":
    pool = Pool(20) # use 20 concurrent connections
    params = izip(urls, repeat(data)) # use the same data for all urls
    for url, content, error in pool.imap_unorderred(fetch, params):
        if error is None:
           print("done: %s: %d" % (url, len(content)))
        else:
           print("error: %s: %s" % (url, error))

503 Service Unavailable is a server error. It might fail to handle the load.

Name or service not known is a dns error. If you need make many requests; install/enable a local caching dns server.

More Related Contents:

  1. Import error: No module name urllib2
  2. Changing user agent on urllib2.urlopen
  3. Stream large binary files with urllib2 to file
  4. How can I use a SOCKS 4/5 proxy with urllib2?
  5. Python: URLError:
  6. Python: download files from google drive using url
  7. Python handling socket.error: [Errno 104] Connection reset by peer
  8. How do I prevent Python’s urllib(2) from following a redirect
  9. Python URLLib / URLLib2 POST
  10. Python: HTTP Post a large file with streaming
  11. python: urllib2 how to send cookie with urlopen request
  12. urllib2 read to Unicode
  13. python ignore certificate validation urllib2
  14. Login to website using urllib2 – Python 2.7
  15. Handling urllib2’s timeout? – Python
  16. Python urllib2.urlopen() is slow, need a better way to read several urls
  17. Python Requests: Post JSON and file in single request
  18. What should I do if socket.setdefaulttimeout() is not working?
  19. Multiple (asynchronous) connections with urllib2 or other http library?
  20. How can I speed up fetching pages with urllib2 in python?
  21. Tell urllib2 to use custom DNS
  22. urllib2.URLError:
  23. opening websites using urllib2 from behind corporate firewall – 11004 getaddrinfo failed
  24. Requests, bind to an ip
  25. Are urllib2 and httplib thread safe?
  26. Making HTTP HEAD request with urllib2 from Python 2
  27. How do I catch a specific HTTP error in Python?
  28. How to get the URL of a redirect with Python
  29. How do I send a custom header with urllib2 in a HTTP Request?
  30. urllib2 and json

Leave a Comment