How to limit concurrency with Python asyncio?

by

If I’m not mistaken you’re searching for asyncio.Semaphore. Example of usage:

import asyncio
from random import randint


async def download(code):
    wait_time = randint(1, 3)
    print('downloading {} will take {} second(s)'.format(code, wait_time))
    await asyncio.sleep(wait_time)  # I/O, context will switch to main function
    print('downloaded {}'.format(code))


sem = asyncio.Semaphore(3)


async def safe_download(i):
    async with sem:  # semaphore limits num of simultaneous downloads
        return await download(i)


async def main():
    tasks = [
        asyncio.ensure_future(safe_download(i))  # creating task starts coroutine
        for i
        in range(9)
    ]
    await asyncio.gather(*tasks)  # await moment all downloads done


if __name__ ==  '__main__':
    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(main())
    finally:
        loop.run_until_complete(loop.shutdown_asyncgens())
        loop.close()

Output:

downloading 0 will take 3 second(s)
downloading 1 will take 3 second(s)
downloading 2 will take 1 second(s)
downloaded 2
downloading 3 will take 3 second(s)
downloaded 1
downloaded 0
downloading 4 will take 2 second(s)
downloading 5 will take 1 second(s)
downloaded 5
downloaded 3
downloading 6 will take 3 second(s)
downloading 7 will take 1 second(s)
downloaded 4
downloading 8 will take 2 second(s)
downloaded 7
downloaded 8
downloaded 6

An example of async downloading with aiohttp can be found here. Note that aiohttp has a Semaphore equivalent built in, which you can see an example of here. It has a default limit of 100 connections.

More Related Contents:

  1. Simplest async/await example possible in Python
  2. How can I wrap a synchronous function in an async coroutine?
  3. asyncio.sleep() vs time.sleep() in Python
  4. how to add a coroutine to a running asyncio loop?
  5. How to combine Celery with asyncio?
  6. How does asyncio actually work?
  7. How can I periodically execute a function with asyncio?
  8. Python 3.7 – asyncio.sleep() and time.sleep()
  9. FastAPI runs api-calls in serial instead of parallel fashion
  10. asyncio.sleep() vs time.sleep()
  11. multiprocessing vs multithreading vs asyncio in Python 3
  12. How to use asyncio with existing blocking library?
  13. Asyncio.gather vs asyncio.wait
  14. Sharing python objects across multiple workers
  15. Can an asyncio event loop run in the background without suspending the Python interpreter?
  16. Combine awaitables like Promise.all
  17. How to test Python 3.4 asyncio code?
  18. Does asyncio supports asynchronous I/O for file operations?
  19. What does “SSLError: [SSL] PEM lib (_ssl.c:2532)” mean using the Python ssl library?
  20. How to run multiple functions at the same time?
  21. How to use threading in Python?
  22. How to call an async function without await?
  23. multiprocessing vs multithreading vs asyncio
  24. Use asyncio and Tkinter (or another GUI lib) together without freezing the GUI
  25. Using asyncio.Queue for producer-consumer flow
  26. asyncio.ensure_future vs. BaseEventLoop.create_task vs. simple coroutine?
  27. How to use await in a python lambda
  28. How to properly create and run concurrent tasks using python’s asyncio module?
  29. Is there a way to use asyncio.Queue in multiple threads?
  30. “asyncio.run() cannot be called from a running event loop” when using Jupyter Notebook

Leave a Comment