Multiprocessing vs Threading Python [duplicate]

by

Here are some pros/cons I came up with.

Multiprocessing

Pros

  • Separate memory space
  • Code is usually straightforward
  • Takes advantage of multiple CPUs & cores
  • Avoids GIL limitations for cPython
  • Eliminates most needs for synchronization primitives unless if you use shared memory (instead, it’s more of a communication model for IPC)
  • Child processes are interruptible/killable
  • Python multiprocessing module includes useful abstractions with an interface much like threading.Thread
  • A must with cPython for CPU-bound processing

Cons

  • IPC a little more complicated with more overhead (communication model vs. shared memory/objects)
  • Larger memory footprint

Threading

Pros

  • Lightweight – low memory footprint
  • Shared memory – makes access to state from another context easier
  • Allows you to easily make responsive UIs
  • cPython C extension modules that properly release the GIL will run in parallel
  • Great option for I/O-bound applications

Cons

  • cPython – subject to the GIL
  • Not interruptible/killable
  • If not following a command queue/message pump model (using the Queue module), then manual use of synchronization primitives become a necessity (decisions are needed for the granularity of locking)
  • Code is usually harder to understand and to get right – the potential for race conditions increases dramatically

More Related Contents:

  1. Can’t pickle when using multiprocessing Pool.map()
  2. How to run functions in parallel?
  3. multiprocessing.Pool: When to use apply, apply_async or map?
  4. What are the differences between the threading and multiprocessing modules?
  5. How to use multiprocessing queue in Python?
  6. Processing single file from multiple processes
  7. multiprocessing vs multithreading vs asyncio in Python 3
  8. What kind of problems (if any) would there be combining asyncio with multiprocessing?
  9. multiprocessing vs multithreading vs asyncio
  10. Using multiprocessing.Process with a maximum number of simultaneous processes
  11. How can I use threading in Python?
  12. Use numpy array in shared memory for multiprocessing
  13. Daemon Threads Explanation
  14. Multiprocessing: How to use Pool.map on a function defined in a class?
  15. Problem with multi threaded Python app and socket connections
  16. Multiprocessing example giving AttributeError
  17. Redirecting stdout and stderr to a PyQt4 QTextEdit from a secondary thread
  18. time.sleep — sleeps thread or process?
  19. How do I get a thread safe print in Python 2.6?
  20. Stopping a thread after a certain amount of time
  21. Multiprocessing.Pool makes Numpy matrix multiplication slower
  22. Multiprocessing causes Python to crash and gives an error may have been in progress in another thread when fork() was called
  23. Python requests with multithreading
  24. Using a global dictionary with threads in Python
  25. Python multiprocessing – tracking the process of pool.map operation
  26. Creating and updating nested dictionaries and lists inside multiprocessing.Manager object
  27. How to obtain a Thread id in Python?
  28. Dump stacktraces of all active Threads
  29. How to use QThread correctly in pyqt with moveToThread()?
  30. ‘sys.excepthook’ and threading

Leave a Comment