multiprocessing vs multithreading vs asyncio

by

TL;DR

Making the Right Choice:

We have walked through the most popular forms of concurrency. But the question remains – when should choose which one? It really depends on the use cases. From my experience (and reading), I tend to follow this pseudo code:

if io_bound:
    if io_very_slow:
        print("Use Asyncio")
    else:
        print("Use Threads")
else:
    print("Multi Processing")
  • CPU Bound => Multi Processing
  • I/O Bound, Fast I/O, Limited Number of Connections => Multi Threading
  • I/O Bound, Slow I/O, Many connections => Asyncio

Reference

[NOTE]:

  • If you have a long call method (e.g. a method containing a sleep time or lazy I/O), the best choice is asyncio, Twisted or Tornado approach (coroutine methods), that works with a single thread as concurrency.
  • asyncio works on Python3.4 and later.
  • Tornado and Twisted are ready since Python2.7
  • uvloop is ultra fast asyncio event loop (uvloop makes asyncio 2-4x faster).

[UPDATE (2019)]:

  • Japranto (GitHub) is a very fast pipelining HTTP server based on uvloop.

More Related Contents:

  1. multiprocessing vs multithreading vs asyncio in Python 3
  2. What kind of problems (if any) would there be combining asyncio with multiprocessing?
  3. How to use asyncio with threading in Python is Method threading.Timer
  4. Multiprocessing vs Threading Python [duplicate]
  5. How does asyncio actually work?
  6. Are lists thread-safe?
  7. multiprocessing: Understanding logic behind `chunksize`
  8. How to limit concurrency with Python asyncio?
  9. How to use multiprocessing queue in Python?
  10. Using win32com with multithreading
  11. asyncio.sleep() vs time.sleep()
  12. Threads and tkinter
  13. How to combine python asyncio with threads?
  14. How to use asyncio with existing blocking library?
  15. Concurrent.futures vs Multiprocessing in Python 3
  16. Simplest async/await example possible in Python
  17. How to test Python 3.4 asyncio code?
  18. How to use threading in Python?
  19. How to call an async function without await?
  20. Python selenium multiprocessing
  21. How to share numpy random state of a parent process with child processes?
  22. Using asyncio.Queue for producer-consumer flow
  23. Trying to fix tkinter GUI freeze-ups (using threads)
  24. What’s the difference between ThreadPool vs Pool in the multiprocessing module?
  25. Python OpenCV multiprocessing cv2.VideoCapture mp4
  26. Using 100% of all cores with the multiprocessing module
  27. How to limit number of CPU’s used by a python script w/o terminal or multiprocessing library?
  28. With “args” vs without “args” to pass arguments to a thread in Python
  29. how to add a coroutine to a running asyncio loop?
  30. How to combine Celery with asyncio?

Leave a Comment