How to parallelize file downloads?

by

You could use a thread pool to download files in parallel:

#!/usr/bin/env python3
from multiprocessing.dummy import Pool # use threads for I/O bound tasks
from urllib.request import urlretrieve

urls = [...]
result = Pool(4).map(urlretrieve, urls) # download 4 files at a time

You could also download several files at once in a single thread using asyncio:

#!/usr/bin/env python3
import asyncio
import logging
from contextlib import closing
import aiohttp # $ pip install aiohttp

@asyncio.coroutine
def download(url, session, semaphore, chunk_size=1<<15):
    with (yield from semaphore): # limit number of concurrent downloads
        filename = url2filename(url)
        logging.info('downloading %s', filename)
        response = yield from session.get(url)
        with closing(response), open(filename, 'wb') as file:
            while True: # save file
                chunk = yield from response.content.read(chunk_size)
                if not chunk:
                    break
                file.write(chunk)
        logging.info('done %s', filename)
    return filename, (response.status, tuple(response.headers.items()))

urls = [...]
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
with closing(asyncio.get_event_loop()) as loop, \
     closing(aiohttp.ClientSession()) as session:
    semaphore = asyncio.Semaphore(4)
    download_tasks = (download(url, session, semaphore) for url in urls)
    result = loop.run_until_complete(asyncio.gather(*download_tasks))

where url2filename() is defined here.

More Related Contents:

  1. Calling a python script with input within a python script using subprocess
  2. Subprocess.Popen: cloning stdout and stderr both to terminal and variables
  3. Python output to Console within Subprocess from the child scricpt
  4. How to suppress or capture the output of subprocess.run()?
  5. Sending a password over SSH or SCP with subprocess.Popen
  6. Trying to fix tkinter GUI freeze-ups (using threads)
  7. What is the use of stub files (.pyi ) in python?
  8. “subprocess.Popen” – checking for success and errors
  9. Subprocess check_output returned non-zero exit status 1
  10. Convert bytes to a string
  11. How do I use raw_input in Python 3
  12. Blocking and Non Blocking subprocess calls
  13. Find all combinations of a list of numbers with a given sum
  14. python ctypes issue on different OSes
  15. Threads and tkinter
  16. numpy is already installed with Anaconda but I get an ImportError (DLL load failed: The specified module could not be found)
  17. real time subprocess.Popen via stdout and PIPE
  18. Interacting with bash from python
  19. How to use threading in Python?
  20. How to check if a variable is a function in Python?
  21. What’s the difference between str.isdigit(), isnumeric() and isdecimal() in Python?
  22. “Failed to load HostKeys” warning while connecting to SFTP server with pysftp
  23. Class Based Views VS Function Based Views
  24. Using 100% of all cores with the multiprocessing module
  25. pass task instance variables as op_args
  26. Changing the directory where .pyc files are created
  27. In Python, when are two objects the same?
  28. Handle JSON Decode Error when nothing returned
  29. ‘is’ operator behaves unexpectedly with floats
  30. Right-to-Left and Left-to-Right printed nicely

Leave a Comment