How to combine python asyncio with threads?

by

It’s pretty simple to delegate a method to a thread or sub-process using BaseEventLoop.run_in_executor:

import asyncio
import time
from concurrent.futures import ProcessPoolExecutor

def cpu_bound_operation(x):
    time.sleep(x) # This is some operation that is CPU-bound

@asyncio.coroutine
def main():
    # Run cpu_bound_operation in the ProcessPoolExecutor
    # This will make your coroutine block, but won't block
    # the event loop; other coroutines can run in meantime.
    yield from loop.run_in_executor(p, cpu_bound_operation, 5)


loop = asyncio.get_event_loop()
p = ProcessPoolExecutor(2) # Create a ProcessPool with 2 processes
loop.run_until_complete(main())

As for whether to use a ProcessPoolExecutor or ThreadPoolExecutor, that’s kind of hard to say; pickling a large object will definitely eat some CPU cycles, which initially would make you think ProcessPoolExecutor is the way to go. However, passing your 100MB object to a Process in the pool would require pickling the instance in your main process, sending the bytes to the child process via IPC, unpickling it in the child, and then pickling it again so you can write it to disk. Given that, my guess is the pickling/unpickling overhead will be large enough that you’re better off using a ThreadPoolExecutor, even though you’re going to take a performance hit because of the GIL.

That said, it’s very simple to test both ways and find out for sure, so you might as well do that.

More Related Contents:

  1. How to use asyncio with threading in Python is Method threading.Timer
  2. multiprocessing vs multithreading vs asyncio in Python 3
  3. How can I call an async function without await?
  4. Learning asyncio: “coroutine was never awaited” warning error
  5. What kind of problems (if any) would there be combining asyncio with multiprocessing?
  6. How can I wrap a synchronous function in an async coroutine?
  7. How to call an async function without await?
  8. multiprocessing vs multithreading vs asyncio
  9. aiohttp: set maximum number of requests per second
  10. aiohttp: rate limiting parallel requests
  11. Exception event loop is closed with aiohttp and asyncio in python 3.8
  12. How can I use threading in Python?
  13. Using module ‘subprocess’ with timeout
  14. How to limit execution time of a function call?
  15. Python threading multiple bash subprocesses?
  16. Make 2 functions run at the same time
  17. Limit number of threads in numpy
  18. How to capture multiple camera streams with OpenCV?
  19. Thread Safety in Python’s dictionary
  20. Threading in Python [closed]
  21. RuntimeError: This event loop is already running in python
  22. Sharing python objects across multiple workers
  23. Does asyncio supports asynchronous I/O for file operations?
  24. Tkinter GUI, I/O & Threading: When to use queues, when events?
  25. Using a global dictionary with threads in Python
  26. Python thread name doesn’t show up on ps or htop
  27. Threads are printing at the same time messing up the text output
  28. Access a QLCDNumber object from a external function
  29. Prompt for user input using python asyncio.create_server instance
  30. waiting for user input in separate thread

Leave a Comment