Python Multiprocess Pool. How to exit the script when one of the worker process determines no more work needs to be done?

by

You can use callbacks from Pool.apply_async.

Something like this should do the job for you.

from multiprocessing import Pool


def part_crack_helper(args):
    solution = do_job(args)
    if solution:
        return True
    else:
        return False


class Worker():
    def __init__(self, workers, initializer, initargs):
        self.pool = Pool(processes=workers, 
                         initializer=initializer, 
                         initargs=initargs)

    def callback(self, result):
        if result:
            print("Solution found! Yay!")
            self.pool.terminate()

    def do_job(self):
        for args in product(seed_str, repeat=4):
            self.pool.apply_async(part_crack_helper, 
                                  args=args, 
                                  callback=self.callback)

        self.pool.close()
        self.pool.join()
        print("good bye")


w = Worker(num_proc, init, [total_count])
w.do_job()

More Related Contents:

  1. Can’t pickle when using multiprocessing Pool.map()
  2. Keyboard Interrupts with python’s multiprocessing Pool
  3. Python Process Pool non-daemonic?
  4. Combine Pool.map with shared memory Array in Python multiprocessing
  5. Filling a queue and managing multiprocessing in python
  6. Multiprocessing.Pool makes Numpy matrix multiplication slower
  7. Can I use a multiprocessing Queue in a function called by Pool.imap?
  8. multiprocessing.Pool() slower than just using ordinary functions
  9. How do you pass a Queue reference to a function managed by pool.map_async()?
  10. Passing multiple parameters to pool.map() function in Python [duplicate]
  11. How to let Pool.map take a lambda function
  12. Python: Writing to a single file with queue while using multiprocessing Pool
  13. Python multiprocessing – tracking the process of pool.map operation
  14. How to use Python multiprocessing Pool.map to fill numpy array in a for loop
  15. How to use multiprocessing pool.map with multiple arguments
  16. What are the differences between the threading and multiprocessing modules?
  17. Is shared readonly data copied to different processes for multiprocessing?
  18. python multiprocessing on windows, if __name__ == “__main__”
  19. Sharing a complex object between processes?
  20. What can multiprocessing and dill do together?
  21. Child processes created with python multiprocessing module won’t print
  22. Python command line input in a process
  23. What exactly is Python multiprocessing Module’s .join() Method Doing?
  24. yet another confusion with multiprocessing error, ‘module’ object has no attribute ‘f’
  25. Catch Ctrl+C / SIGINT and exit multiprocesses gracefully in python [duplicate]
  26. Multiprocessing launching too many instances of Python VM
  27. Modify object in python multiprocessing
  28. multiprocessing pool example does not work and freeze the kernel
  29. Designate specific CPU for a process – python multiprocessing
  30. Using python multiprocessing with different random seed for each process

Leave a Comment