Script using multiprocessing module does not terminate

by

This is because when you put lots of items into a multiprocessing.Queue, they eventually get buffered in memory, once the underlying Pipe is full. The buffer won’t get flushed until something starts reading from the other end of the Queue, which will allow the Pipe to accept more data. A Process cannot terminate until the buffer for all its Queue instances have been entirely flushed to their underlying Pipe. The implication of this is that if you try to join a process without having another process/thread calling get on its Queue, you could deadlock. This is mentioned in the docs:

Warning

As mentioned above, if a child process has put items on a queue (and
it has not used JoinableQueue.cancel_join_thread), then that process
will not terminate until all buffered items have been flushed to the
pipe.

This means that if you try joining that process you may get a deadlock
unless you are sure that all items which have been put on the queue
have been consumed. Similarly, if the child process is non-daemonic
then the parent process may hang on exit when it tries to join all its
non-daemonic children.

Note that a queue created using a manager does not have this issue.

You can fix the issue by not calling join until after you empty the Queue in the parent:

for _ in xrange(len(langs)):
    item = que.get()
    print(item)
    dicList.append(item)

# join after emptying the queue.
for p in processList:
    p.join()

print("here")

More Related Contents:

  1. How to use multiprocessing pool.map with multiple arguments
  2. Python multiprocessing PicklingError: Can’t pickle
  3. Sharing a result queue among several processes
  4. multiprocessing: Understanding logic behind `chunksize`
  5. Dead simple example of using Multiprocessing Queue, Pool and Locking
  6. Shared variable in python’s multiprocessing
  7. multiprocessing.Pool: What’s the difference between map_async and imap?
  8. Why does Python’s multiprocessing module import __main__ when starting a new process on Windows?
  9. Why can I pass an instance method to multiprocessing.Process, but not a multiprocessing.Pool?
  10. multiprocessing.pool.MaybeEncodingError: ‘TypeError(“cannot serialize ‘_io.BufferedReader’ object”,)’
  11. Leveraging “Copy-on-Write” to Copy Data to Multiprocessing.Pool() Worker Processes
  12. Python multiprocessing.Pool: AttributeError
  13. How to get the return value of a function passed to multiprocessing.Process?
  14. What’s the difference between ThreadPool vs Pool in the multiprocessing module?
  15. how to to terminate process using python’s multiprocessing
  16. multiprocessing pool example does not work and freeze the kernel
  17. Python Multiprocessing error: AttributeError: module ‘__main__’ has no attribute ‘__spec__’
  18. Multiprocessing AsyncResult.get() hangs in Python 3.7.2 but not in 3.6
  19. Creating and updating nested dictionaries and lists inside multiprocessing.Manager object
  20. Python multiprocessing application is getting stuck in docker container
  21. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  22. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  23. How can I abort a task in a multiprocessing.Pool after a timeout?
  24. How can I share a class between processes?
  25. How to limit number of CPU’s used by a python script w/o terminal or multiprocessing library?
  26. Numpy – create matrix with rows of vector
  27. Can json.loads ignore trailing commas?
  28. Sending a file over TCP sockets in Python
  29. Python 2: SMTPServerDisconnected: Connection unexpectedly closed
  30. How can I import files in Python using sys.path.append?

Leave a Comment