Python 3 Multiprocessing queue deadlock when calling join before the queue is empty

by

The queue implementation in multiprocessing that allows data to be transferred between processes relies on standard OS pipes.

OS pipes are not infinitely long, so the process which queues data could be blocked in the OS during the put() operation until some other process uses get() to retrieve data from the queue.

For small amounts of data, such as the one in your example, the main process can join() all the spawned subprocesses and then pick up the data. This often works well, but does not scale, and it is not clear when it will break.

But it will certainly break with large amounts of data. The subprocess will be blocked in put() waiting for the main process to remove some data from the queue with get(), but the main process is blocked in join() waiting for the subprocess to finish. This results in a deadlock.

Here is an example where a user had this exact issue. I posted some code in an answer there that helped him solve his problem.

More Related Contents:

  1. Timeout on a function call
  2. Tkinter: How to use threads to preventing main event loop from “freezing”
  3. Can’t pickle when using multiprocessing Pool.map()
  4. Background thread with QThread in PyQt
  5. Run certain code every n seconds [duplicate]
  6. What are the differences between the threading and multiprocessing modules?
  7. Creating Threads in python
  8. thread starts running before calling Thread.start
  9. How to use multiprocessing queue in Python?
  10. PyQt4 Wait in thread for user input from GUI
  11. How to terminate a thread when main program ends?
  12. Execute a function after Flask returns response
  13. Thread that I can pause and resume?
  14. python multi-threading slower than serial?
  15. What is “thread local storage” in Python, and why do I need it?
  16. why isn’t numpy.mean multithreaded?
  17. Python Threading inside a class
  18. Meaning of daemon property on Python Threads
  19. How to get the return value from a thread?
  20. Running a Tkinter form in a separate thread
  21. Threaded Tkinter script crashes when creating the second Toplevel widget
  22. Are Generators Threadsafe?
  23. Is Keras thread safe?
  24. Python Tkinter Label redrawing every 10 seconds
  25. threading appears to run threads sequentially
  26. How do I use threading in Python?
  27. With “args” vs without “args” to pass arguments to a thread in Python
  28. How can I profile a multithread program in Python?
  29. How to prefetch data using a custom python function in tensorflow
  30. How do I run a long-running job in the background in Python

Leave a Comment