Fetching multiple urls with aiohttp in python

by

Working example: 

import asyncio
import aiohttp
import ssl

url_list = ['https://api.pushshift.io/reddit/search/comment/?q=Nestle&size=30&after=1530396000&before=1530436000',
            'https://api.pushshift.io/reddit/search/comment/?q=Nestle&size=30&after=1530436000&before=1530476000']


async def fetch(session, url):
    async with session.get(url, ssl=ssl.SSLContext()) as response:
        return await response.json()


async def fetch_all(urls, loop):
    async with aiohttp.ClientSession(loop=loop) as session:
        results = await asyncio.gather(*[fetch(session, url) for url in urls], return_exceptions=True)
        return results


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    urls = url_list
    htmls = loop.run_until_complete(fetch_all(urls, loop))
    print(htmls)

More Related Contents:

  1. Wait page to load before getting data with requests.get in python 3
  2. Failed to load resource: the server responded with a status of 429 (Too Many Requests) and 404 (Not Found) with ChromeDriver Chrome through Selenium
  3. How can I wrap a synchronous function in an async coroutine?
  4. “Eager” Page Load Strategy workaround for Chromedriver Selenium in Python
  5. How to call an async function without await?
  6. Python async: Waiting for stdin input while doing other stuff
  7. How i can get new ip from tor every requests in threads?
  8. using Subprocess to avoid long-running task from disconnecting discord.py bot?
  9. How to Convert python list into nested dictionaries in pythonic way [closed]
  10. Writing a specific function in python
  11. Invalid selector: Compound class names not permitted error using Selenium
  12. webdriver.FirefoxProfile(): Is it possible to use a profile without making a copy of it?
  13. No multiprocessing print outputs (Spyder)
  14. Clear PyCharm Run Window
  15. How to use Tor with Chrome browser through Selenium
  16. Selenium “Unable to find a matching set of capabilities” despite driver being in /usr/local/bin
  17. ‘python3’ is not recognized as an internal or external command, operable program or batch file
  18. ModuleNotFoundError: No module named ‘__main__.xxxx’; ‘__main__’ is not a package
  19. How to set class attribute with await in __init__
  20. aiohttp: set maximum number of requests per second
  21. How can I await inside future-like object’s __await__?
  22. Tkinter: How to make a rounded corner text widget?
  23. pandas multiindex – how to select second level when using columns?
  24. How to suppress console error/warning/info messages when executing selenium python scripts using chrome canary
  25. Is there a way to use asyncio.Queue in multiple threads?
  26. How to create Python secure websocket client request?
  27. ImportError: cannot import name ‘escape’ from ‘jinja2’
  28. Airflow s3 connection using UI
  29. Can’t install Matplotlib on Python 3.10 after its release (2021-10-05)
  30. How to combine Celery with asyncio?

Leave a Comment