Python Process Pool non-daemonic?

by

The multiprocessing.pool.Pool class creates the worker processes in its __init__ method, makes them daemonic and starts them, and it is not possible to re-set their daemon attribute to False before they are started (and afterwards it’s not allowed anymore). But you can create your own sub-class of multiprocesing.pool.Pool (multiprocessing.Pool is just a wrapper function) and substitute your own multiprocessing.Process sub-class, which is always non-daemonic, to be used for the worker processes.

Here’s a full example of how to do this. The important parts are the two classes NoDaemonProcess and MyPool at the top and to call pool.close() and pool.join() on your MyPool instance at the end.

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

import multiprocessing
# We must import this explicitly, it is not imported by the top-level
# multiprocessing module.
import multiprocessing.pool
import time

from random import randint


class NoDaemonProcess(multiprocessing.Process):
    # make 'daemon' attribute always return False
    def _get_daemon(self):
        return False
    def _set_daemon(self, value):
        pass
    daemon = property(_get_daemon, _set_daemon)

# We sub-class multiprocessing.pool.Pool instead of multiprocessing.Pool
# because the latter is only a wrapper function, not a proper class.
class MyPool(multiprocessing.pool.Pool):
    Process = NoDaemonProcess

def sleepawhile(t):
    print("Sleeping %i seconds..." % t)
    time.sleep(t)
    return t

def work(num_procs):
    print("Creating %i (daemon) workers and jobs in child." % num_procs)
    pool = multiprocessing.Pool(num_procs)

    result = pool.map(sleepawhile,
        [randint(1, 5) for x in range(num_procs)])

    # The following is not really needed, since the (daemon) workers of the
    # child's pool are killed when the child is terminated, but it's good
    # practice to cleanup after ourselves anyway.
    pool.close()
    pool.join()
    return result

def test():
    print("Creating 5 (non-daemon) workers and jobs in main process.")
    pool = MyPool(5)

    result = pool.map(work, [randint(1, 5) for x in range(5)])

    pool.close()
    pool.join()
    print(result)

if __name__ == '__main__':
    test()

More Related Contents:

  1. Can’t pickle when using multiprocessing Pool.map()
  2. Keyboard Interrupts with python’s multiprocessing Pool
  3. Combine Pool.map with shared memory Array in Python multiprocessing
  4. Filling a queue and managing multiprocessing in python
  5. Multiprocessing.Pool makes Numpy matrix multiplication slower
  6. Can I use a multiprocessing Queue in a function called by Pool.imap?
  7. multiprocessing.Pool() slower than just using ordinary functions
  8. How do you pass a Queue reference to a function managed by pool.map_async()?
  9. Passing multiple parameters to pool.map() function in Python [duplicate]
  10. Python Multiprocess Pool. How to exit the script when one of the worker process determines no more work needs to be done?
  11. How to let Pool.map take a lambda function
  12. Python: Writing to a single file with queue while using multiprocessing Pool
  13. Python multiprocessing – tracking the process of pool.map operation
  14. How to use Python multiprocessing Pool.map to fill numpy array in a for loop
  15. multiprocessing: How do I share a dict among multiple processes?
  16. multiprocessing: sharing a large read-only object between processes?
  17. Python multiprocessing safely writing to a file
  18. Compulsory usage of if __name__==”__main__” in windows while using multiprocessing [duplicate]
  19. Multiprocessing – Pipe vs Queue
  20. Concurrent.futures vs Multiprocessing in Python 3
  21. Parallelizing a Numpy vector operation
  22. multiprocessing.pool.MaybeEncodingError: ‘TypeError(“cannot serialize ‘_io.BufferedReader’ object”,)’
  23. High Memory Usage Using Python Multiprocessing
  24. OSError 38 [Errno 38] with multiprocessing
  25. Can multiprocessing Process class be run from IDLE
  26. Python multiprocess profiling
  27. What’s the difference between ThreadPool vs Pool in the multiprocessing module?
  28. Using 100% of all cores with the multiprocessing module
  29. Mix Python Twisted with multiprocessing?
  30. Using multiprocessing.Process with a maximum number of simultaneous processes

Leave a Comment