When should we call multiprocessing.Pool.join?

by

No, you don’t, but it’s probably a good idea if you aren’t going to use the pool anymore.

Reasons for calling pool.close or pool.join are well said by Tim Peters in this SO post:

As to Pool.close(), you should call that when – and only when – you’re never going to submit more work to the Pool instance. So Pool.close() is typically called when the parallelizable part of your main program is finished. Then the worker processes will terminate when all work already assigned has completed.

It’s also excellent practice to call Pool.join() to wait for the worker processes to terminate. Among other reasons, there’s often no good way to report exceptions in parallelized code (exceptions occur in a context only vaguely related to what your main program is doing), and Pool.join() provides a synchronization point that can report some exceptions that occurred in worker processes that you’d otherwise never see.

More Related Contents:

  1. How to use multiprocessing pool.map with multiple arguments
  2. Python multiprocessing PicklingError: Can’t pickle
  3. How can I recover the return value of a function passed to multiprocessing.Process?
  4. Sharing a result queue among several processes
  5. multiprocessing: Understanding logic behind `chunksize`
  6. Shared variable in python’s multiprocessing
  7. multiprocessing.Pool: What’s the difference between map_async and imap?
  8. Script using multiprocessing module does not terminate
  9. Keras + Tensorflow and Multiprocessing in Python
  10. Is it possible to run function in a subprocess without threading or writing a separate file/script.
  11. Preserve custom attributes when pickling subclass of numpy array
  12. multiprocessing.pool.MaybeEncodingError: ‘TypeError(“cannot serialize ‘_io.BufferedReader’ object”,)’
  13. Leveraging “Copy-on-Write” to Copy Data to Multiprocessing.Pool() Worker Processes
  14. Python multiprocessing.Pool: AttributeError
  15. How to change the serialization method used by the multiprocessing module?
  16. Using stdin in a child Process
  17. Processes stuck in loop with PyInstaller-executable
  18. How to get the return value of a function passed to multiprocessing.Process?
  19. What’s the difference between ThreadPool vs Pool in the multiprocessing module?
  20. how to to terminate process using python’s multiprocessing
  21. multiprocessing pool example does not work and freeze the kernel
  22. Python Multiprocessing error: AttributeError: module ‘__main__’ has no attribute ‘__spec__’
  23. Creating and updating nested dictionaries and lists inside multiprocessing.Manager object
  24. Python multiprocessing application is getting stuck in docker container
  25. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  26. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  27. Multiprocessing: use only the physical cores?
  28. How can I share a class between processes?
  29. How to limit number of CPU’s used by a python script w/o terminal or multiprocessing library?
  30. cProfile causes pickling error when running multiprocessing Python code

Leave a Comment