Python: Getting a traceback from a multiprocessing.Process

by

Using tblib you can pass wrapped exceptions and reraise them later:

import tblib.pickling_support
tblib.pickling_support.install()

from multiprocessing import Pool
import sys


class ExceptionWrapper(object):

    def __init__(self, ee):
        self.ee = ee
        __, __, self.tb = sys.exc_info()

    def re_raise(self):
        raise self.ee.with_traceback(self.tb)
        # for Python 2 replace the previous line by:
        # raise self.ee, None, self.tb


# example of how to use ExceptionWrapper

def inverse(i):
    """ will fail for i == 0 """
    try:
        return 1.0 / i
    except Exception as e:
        return ExceptionWrapper(e)


def main():
    p = Pool(1)
    results = p.map(inverse, [0, 1, 2, 3])
    for result in results:
        if isinstance(result, ExceptionWrapper):
            result.re_raise()


if __name__ == "__main__":
    main()

So, if you catch an exception in your remote process, wrap it with ExceptionWrapper and then pass it back. Calling re_raise() in the main process will do the work.

More Related Contents:

  1. What are the differences between the threading and multiprocessing modules?
  2. How to catch and print the full exception traceback without halting/exiting the program?
  3. When I catch an exception, how do I get the type, file, and line number?
  4. Sharing a complex object between processes?
  5. Exception thrown in multiprocessing Pool not detected
  6. Simple Python Multiprocessing function in Spyder doesn’t output results
  7. How can I more easily suppress previous exceptions when I raise my own exception in response?
  8. How to get the return value of a function passed to multiprocessing.Process?
  9. Print an error message without printing a traceback and close the program when a condition is not met
  10. Catch and print full Python exception traceback without halting/exiting the program
  11. Why does multiprocessing use only a single core after I import numpy?
  12. Sharing a result queue among several processes
  13. Keyboard Interrupts with python’s multiprocessing Pool
  14. multiprocessing: Understanding logic behind `chunksize`
  15. How to increment a shared counter from multiple processes?
  16. multiprocessing.dummy in Python is not utilising 100% cpu
  17. Same output in different workers in multiprocessing
  18. How to get exception message in Python properly
  19. How to add a timeout to a function in Python
  20. Multiprocessing working in Python but not in iPython
  21. Terminate a python script from another python script
  22. How to share numpy random state of a parent process with child processes?
  23. multiprocessing vs multithreading vs asyncio
  24. hasattr() vs try-except block to deal with non-existent attributes
  25. How to re-raise an exception in nested try/except blocks?
  26. Get Traceback of warnings
  27. Python multiprocessing – tracking the process of pool.map operation
  28. Creating and updating nested dictionaries and lists inside multiprocessing.Manager object
  29. Using multiprocessing.Process with a maximum number of simultaneous processes
  30. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python

Leave a Comment