Catch Ctrl+C / SIGINT and exit multiprocesses gracefully in python [duplicate]

by

The previously accepted solution has race conditions and it does not work with map and async functions.

The correct way to handle Ctrl+C/SIGINT with multiprocessing.Pool is to:

  1. Make the process ignore SIGINT before a process Pool is created. This way created child processes inherit SIGINT handler.
  2. Restore the original SIGINT handler in the parent process after a Pool has been created.
  3. Use map_async and apply_async instead of blocking map and apply.
  4. Wait on the results with timeout because the default blocking waits to ignore all signals. This is Python bug https://bugs.python.org/issue8296.

Putting it together:

#!/bin/env python
from __future__ import print_function

import multiprocessing
import os
import signal
import time

def run_worker(delay):
    print("In a worker process", os.getpid())
    time.sleep(delay)

def main():
    print("Initializng 2 workers")
    original_sigint_handler = signal.signal(signal.SIGINT, signal.SIG_IGN)
    pool = multiprocessing.Pool(2)
    signal.signal(signal.SIGINT, original_sigint_handler)
    try:
        print("Starting 2 jobs of 5 seconds each")
        res = pool.map_async(run_worker, [5, 5])
        print("Waiting for results")
        res.get(60) # Without the timeout this blocking call ignores all signals.
    except KeyboardInterrupt:
        print("Caught KeyboardInterrupt, terminating workers")
        pool.terminate()
    else:
        print("Normal termination")
        pool.close()
    pool.join()

if __name__ == "__main__":
    main()

As @YakovShklarov noted, there is a window of time between ignoring the signal and unignoring it in the parent process, during which the signal can be lost. Using pthread_sigmask instead to temporarily block the delivery of the signal in the parent process would prevent the signal from being lost, however, it is not available in Python-2.

More Related Contents:

  1. How to signal slots in a GUI from a different process?
  2. How to run functions in parallel?
  3. Dead simple example of using Multiprocessing Queue, Pool and Locking
  4. Share Large, Read-Only Numpy Array Between Multiprocessing Processes
  5. multiprocessing.Pool: What’s the difference between map_async and imap?
  6. Python Multiprocessing: Handling Child Errors in Parent
  7. Processing single file from multiple processes
  8. Log output of multiprocessing.Process
  9. pandas multiprocessing apply
  10. Why does Python’s multiprocessing module import __main__ when starting a new process on Windows?
  11. Appending to the same list from different processes using multiprocessing
  12. Exception thrown in multiprocessing Pool not detected
  13. Using the multiprocessing module for cluster computing
  14. multiprocessing.Pool() slower than just using ordinary functions
  15. How do you pass a Queue reference to a function managed by pool.map_async()?
  16. Django: signal when user logs in?
  17. Chrome crashes after several hours while multiprocessing using Selenium through Python
  18. What kind of problems (if any) would there be combining asyncio with multiprocessing?
  19. What is the difference between multiprocessing and subprocess?
  20. Python Multiprocessing Lib Error (AttributeError: __exit__)
  21. Kill or terminate subprocess when timeout?
  22. Show the progress of a Python multiprocessing pool imap_unordered call?
  23. Easiest way to read csv files with multiprocessing in Pandas
  24. Using multiprocessing.Process with a maximum number of simultaneous processes
  25. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  26. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  27. How can I abort a task in a multiprocessing.Pool after a timeout?
  28. Terminate a Python multiprocessing program once a one of its workers meets a certain condition
  29. Understanding scipy deconvolve
  30. Multiprocessing Share Unserializable Objects Between Processes

Leave a Comment