How to properly create and run concurrent tasks using python’s asyncio module?

by

Yes, any coroutine that’s running inside your event loop will block other coroutines and tasks from running, unless it

  1. Calls another coroutine using yield from or await (if using Python 3.5+).
  2. Returns.

This is because asyncio is single-threaded; the only way for the event loop to run is for no other coroutine to be actively executing. Using yield from/await suspends the coroutine temporarily, giving the event loop a chance to work.

Your example code is fine, but in many cases, you probably wouldn’t want long-running code that isn’t doing asynchronous I/O running inside the event loop to begin with. In those cases, it often makes more sense to use asyncio.loop.run_in_executor to run the code in a background thread or process. ProcessPoolExecutor would be the better choice if your task is CPU-bound, ThreadPoolExecutor would be used if you need to do some I/O that isn’t asyncio-friendly.

Your two loops, for example, are completely CPU-bound and don’t share any state, so the best performance would come from using ProcessPoolExecutor to run each loop in parallel across CPUs:

import asyncio
from concurrent.futures import ProcessPoolExecutor

print('running async test')

def say_boo():
    i = 0
    while True:
        print('...boo {0}'.format(i))
        i += 1


def say_baa():
    i = 0
    while True:
        print('...baa {0}'.format(i))
        i += 1

if __name__ == "__main__":
    executor = ProcessPoolExecutor(2)
    loop = asyncio.new_event_loop()
    boo = loop.run_in_executor(executor, say_boo)
    baa = loop.run_in_executor(executor, say_baa)

    loop.run_forever()

More Related Contents:

  1. How to limit concurrency with Python asyncio?
  2. Can an asyncio event loop run in the background without suspending the Python interpreter?
  3. What’s the correct way to clean up after an interrupted event loop?
  4. Why does the asyncio’s event loop suppress the KeyboardInterrupt on Windows?
  5. Python3: ImportError: No module named ‘_ctypes’ when using Value from module multiprocessing
  6. How can I periodically execute a function with asyncio?
  7. Python 3.7 – asyncio.sleep() and time.sleep()
  8. Log output of multiprocessing.Process
  9. RuntimeError: This event loop is already running in python
  10. Python type hinting without cyclic imports
  11. When using asyncio, how do you allow all running tasks to finish before shutting down the event loop
  12. Sharing python objects across multiple workers
  13. Why is one class variable not defined in list comprehension but another is?
  14. How to get text from span tag in BeautifulSoup
  15. Learning asyncio: “coroutine was never awaited” warning error
  16. How can I wrap a synchronous function in an async coroutine?
  17. Does asyncio supports asynchronous I/O for file operations?
  18. What does “SSLError: [SSL] PEM lib (_ssl.c:2532)” mean using the Python ssl library?
  19. How to run multiple functions at the same time?
  20. asyncio.sleep() vs time.sleep() in Python
  21. multiprocessing vs multithreading vs asyncio
  22. Use asyncio and Tkinter (or another GUI lib) together without freezing the GUI
  23. asyncio.ensure_future vs. BaseEventLoop.create_task vs. simple coroutine?
  24. How do you change the value of one attribute by changing the value of another? (dependent attributes)
  25. How do I run two python loops concurrently?
  26. Is there a way to use asyncio.Queue in multiple threads?
  27. “asyncio.run() cannot be called from a running event loop” when using Jupyter Notebook
  28. aiohttp: rate limiting parallel requests
  29. Exception event loop is closed with aiohttp and asyncio in python 3.8
  30. Sharing a lock between gunicorn workers

Leave a Comment