Multiple (asynchronous) connections with urllib2 or other http library?

by

So, it’s 2016 😉 and we have Python 3.4+ with built-in asyncio module for asynchronous I/O. We can use aiohttp as HTTP client to download multiple URLs in parallel.

import asyncio
from aiohttp import ClientSession

async def fetch(url):
    async with ClientSession() as session:
        async with session.get(url) as response:
            return await response.read()

async def run(loop, r):
    url = "http://localhost:8080/{}"
    tasks = []
    for i in range(r):
        task = asyncio.ensure_future(fetch(url.format(i)))
        tasks.append(task)

    responses = await asyncio.gather(*tasks)
    # you now have all response bodies in this variable
    print(responses)

loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run(loop, 4))
loop.run_until_complete(future)

Source: copy-pasted from http://pawelmhm.github.io/asyncio/python/aiohttp/2016/04/22/asyncio-aiohttp.html

More Related Contents:

  1. How to download image using requests
  2. Asynchronous Requests with Python requests
  3. Downloading a picture via urllib and python
  4. Proxy with urllib2
  5. Using an HTTP PROXY – Python [duplicate]
  6. How to fetch a non-ascii url with urlopen?
  7. Scrapy Crawl URLs in Order
  8. Subclassing int in Python
  9. Python Urllib2 SSL error
  10. Using MultipartPostHandler to POST form-data with Python
  11. FastAPI runs api-calls in serial instead of parallel fashion
  12. Python urllib2 Basic Auth Problem
  13. Python-Requests close http connection
  14. Read data from CSV file and transform from string to correct data-type, including a list-of-integer column
  15. Sort nested dictionary by value, and remainder by another value, in Python
  16. How to add a timeout to a function in Python
  17. Python urllib2 Progress Hook
  18. Extracting text from script tag using BeautifulSoup in Python
  19. Simplest async/await example possible in Python
  20. Making a POST call instead of GET using urllib2
  21. What does from __future__ import absolute_import actually do?
  22. How to download any(!) webpage with correct charset in python?
  23. Python requests with multithreading
  24. Python urllib2: Receive JSON response from url
  25. How to use await in a python lambda
  26. Download and decompress gzipped file in memory?
  27. urllib2 file name
  28. How to save “complete webpage” not just basic html using Python
  29. urllib2 and json
  30. how to add a coroutine to a running asyncio loop?

Leave a Comment