Filling a queue and managing multiprocessing in python

by

You could use the blocking capabilities of queue to spawn multiple process at startup (using multiprocessing.Pool) and letting them sleep until some data are available on the queue to process. If your not familiar with that, you could try to “play” with that simple program:

import multiprocessing
import os
import time

the_queue = multiprocessing.Queue()


def worker_main(queue):
    print os.getpid(),"working"
    while True:
        item = queue.get(True)
        print os.getpid(), "got", item
        time.sleep(1) # simulate a "long" operation

the_pool = multiprocessing.Pool(3, worker_main,(the_queue,))
#                           don't forget the comma here  ^

for i in range(5):
    the_queue.put("hello")
    the_queue.put("world")


time.sleep(10)

Tested with Python 2.7.3 on Linux

This will spawn 3 processes (in addition of the parent process). Each child executes the worker_main function. It is a simple loop getting a new item from the queue on each iteration. Workers will block if nothing is ready to process.

At startup all 3 process will sleep until the queue is fed with some data. When a data is available one of the waiting workers get that item and starts to process it. After that, it tries to get an other item from the queue, waiting again if nothing is available…

More Related Contents:

  1. Can I use a multiprocessing Queue in a function called by Pool.imap?
  2. How do you pass a Queue reference to a function managed by pool.map_async()?
  3. Python: Writing to a single file with queue while using multiprocessing Pool
  4. Can’t pickle when using multiprocessing Pool.map()
  5. Sharing a result queue among several processes
  6. Keyboard Interrupts with python’s multiprocessing Pool
  7. Python Process Pool non-daemonic?
  8. Combine Pool.map with shared memory Array in Python multiprocessing
  9. Multiprocessing.Pool makes Numpy matrix multiplication slower
  10. Multiprocessing – Pipe vs Queue
  11. multiprocessing.Pool() slower than just using ordinary functions
  12. Passing multiple parameters to pool.map() function in Python [duplicate]
  13. Python Multiprocess Pool. How to exit the script when one of the worker process determines no more work needs to be done?
  14. How to let Pool.map take a lambda function
  15. python multiprocessing: some functions do not return when they are complete (queue material too big)
  16. Python multiprocessing – tracking the process of pool.map operation
  17. How to use Python multiprocessing Pool.map to fill numpy array in a for loop
  18. How to use multiprocessing pool.map with multiple arguments
  19. What are the differences between the threading and multiprocessing modules?
  20. python multiprocessing on windows, if __name__ == “__main__”
  21. Sharing a complex object between processes?
  22. Script using multiprocessing module does not terminate
  23. Python command line input in a process
  24. yet another confusion with multiprocessing error, ‘module’ object has no attribute ‘f’
  25. Multiprocessing Bomb
  26. Simple Python Multiprocessing function in Spyder doesn’t output results
  27. Python multiprocessing example not working
  28. Multiprocessing launching too many instances of Python VM
  29. Modify object in python multiprocessing
  30. multiprocessing pool example does not work and freeze the kernel

Leave a Comment