What’s the difference between ThreadPool vs Pool in the multiprocessing module?

by

The multiprocessing.pool.ThreadPool behaves the same as the multiprocessing.Pool with the only difference that uses threads instead of processes to run the workers logic.

The reason you see

hi outside of main()

being printed multiple times with the multiprocessing.Pool is due to the fact that the pool will spawn 5 independent processes. Each process will initialize its own Python interpreter and load the module resulting in the top level print being executed again.

Note that this happens only if the spawn process creation method is used (only method available on Windows). If you use the fork one (Unix), you will see the message printed only once as for the threads.

The multiprocessing.pool.ThreadPool is not documented as its implementation has never been completed. It lacks tests and documentation. You can see its implementation in the source code.

I believe the next natural question is: when to use a thread based pool and when to use a process based one?

The rule of thumb is:

  • IO bound jobs -> multiprocessing.pool.ThreadPool
  • CPU bound jobs -> multiprocessing.Pool
  • Hybrid jobs -> depends on the workload, I usually prefer the multiprocessing.Pool due to the advantage process isolation brings

On Python 3 you might want to take a look at the concurrent.future.Executor pool implementations.

More Related Contents:

  1. multiprocessing: Understanding logic behind `chunksize`
  2. Multiprocessing AsyncResult.get() hangs in Python 3.7.2 but not in 3.6
  3. How to limit number of CPU’s used by a python script w/o terminal or multiprocessing library?
  4. How to use multiprocessing pool.map with multiple arguments
  5. Python multiprocessing PicklingError: Can’t pickle
  6. Sharing a result queue among several processes
  7. Shared variable in python’s multiprocessing
  8. multiprocessing.Pool: What’s the difference between map_async and imap?
  9. Script using multiprocessing module does not terminate
  10. multiprocessing vs multithreading vs asyncio in Python 3
  11. Concurrent.futures vs Multiprocessing in Python 3
  12. multiprocessing.pool.MaybeEncodingError: ‘TypeError(“cannot serialize ‘_io.BufferedReader’ object”,)’
  13. Leveraging “Copy-on-Write” to Copy Data to Multiprocessing.Pool() Worker Processes
  14. Python multiprocessing.Pool: AttributeError
  15. How to get the return value of a function passed to multiprocessing.Process?
  16. Python selenium multiprocessing
  17. How to share numpy random state of a parent process with child processes?
  18. multiprocessing.Pool – PicklingError: Can’t pickle : attribute lookup thread.lock failed
  19. multiprocessing vs multithreading vs asyncio
  20. Python OpenCV multiprocessing cv2.VideoCapture mp4
  21. how to to terminate process using python’s multiprocessing
  22. multiprocessing pool example does not work and freeze the kernel
  23. Using 100% of all cores with the multiprocessing module
  24. Python Multiprocessing error: AttributeError: module ‘__main__’ has no attribute ‘__spec__’
  25. Creating and updating nested dictionaries and lists inside multiprocessing.Manager object
  26. Python multiprocessing application is getting stuck in docker container
  27. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  28. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  29. How can I abort a task in a multiprocessing.Pool after a timeout?
  30. How can I share a class between processes?

Leave a Comment