What exactly is Python multiprocessing Module’s .join() Method Doing?

by

The join() method, when used with threading or multiprocessing, is not related to str.join() – it’s not actually concatenating anything together. Rather, it just means “wait for this [thread/process] to complete”. The name join is used because the multiprocessing module’s API is meant to look as similar to the threading module’s API, and the threading module uses join for its Thread object. Using the term join to mean “wait for a thread to complete” is common across many programming languages, so Python just adopted it as well.

Now, the reason you see the 20 second delay both with and without the call to join() is because by default, when the main process is ready to exit, it will implicitly call join() on all running multiprocessing.Process instances. This isn’t as clearly stated in the multiprocessing docs as it should be, but it is mentioned in the Programming Guidelines section:

Remember also that non-daemonic processes will be automatically be
joined.

You can override this behavior by setting the daemon flag on the Process to True prior to starting the process:

p = Process(target=say_hello)
p.daemon = True
p.start()
# Both parent and child will exit here, since the main process has completed.

If you do that, the child process will be terminated as soon as the main process completes:

daemon

The process’s daemon flag, a Boolean value. This must be set before
start() is called.

The initial value is inherited from the creating process.

When a process exits, it attempts to terminate all of its daemonic
child processes.

More Related Contents:

  1. Multiprocessing vs Threading Python [duplicate]
  2. RuntimeError on windows trying python multiprocessing
  3. multiprocessing: How do I share a dict among multiple processes?
  4. multiprocessing: sharing a large read-only object between processes?
  5. Python sharing a lock between processes
  6. How to use multiprocessing queue in Python?
  7. where to put freeze_support() in a Python script?
  8. Multiprocessing a for loop?
  9. Using Python’s Multiprocessing module to execute simultaneous and separate SEAWAT/MODFLOW model runs
  10. Multiprocessing – Pipe vs Queue
  11. Poor scaling of multiprocessing Pool.map() on a list of large objects: How to achieve better parallel scaling in python?
  12. Concurrent.futures vs Multiprocessing in Python 3
  13. Parallelizing a Numpy vector operation
  14. multiprocessing.pool.MaybeEncodingError: ‘TypeError(“cannot serialize ‘_io.BufferedReader’ object”,)’
  15. High Memory Usage Using Python Multiprocessing
  16. “chunksize” parameter in multiprocessing.Pool.map
  17. Connection problems with SQLAlchemy and multiple processes
  18. OSError 38 [Errno 38] with multiprocessing
  19. Can multiprocessing Process class be run from IDLE
  20. Python multiprocess profiling
  21. What’s the difference between ThreadPool vs Pool in the multiprocessing module?
  22. Globals variables and Python multiprocessing [duplicate]
  23. Using 100% of all cores with the multiprocessing module
  24. Mix Python Twisted with multiprocessing?
  25. Using multiprocessing.Process with a maximum number of simultaneous processes
  26. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  27. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  28. How can I abort a task in a multiprocessing.Pool after a timeout?
  29. Terminate a Python multiprocessing program once a one of its workers meets a certain condition
  30. Multiprocessing Share Unserializable Objects Between Processes

Leave a Comment