Python Multiprocessing Lib Error (AttributeError: __exit__)

by

In Python 2.x and 3.0, 3.1 and 3.2, multiprocessing.Pool() objects are not context managers. You cannot use them in a with statement. Only in Python 3.3 and up can you use them as such. From the Python 3 multiprocessing.Pool() documentation:

New in version 3.3: Pool objects now support the context management protocol – see Context Manager Types. __enter__() returns the pool object, and __exit__() calls terminate().

For earlier Python versions, you could use contextlib.closing(), but take into account this’ll call pool.close(), not pool.terminate(). Terminate manually in that case:

from contextlib import closing

with closing(Pool(processes=2)) as pool:
    pool.map(myFunction, mylist)
    pool.map(myfunction2, mylist2)
    pool.terminate()

or create your own terminating() context manager:

from contextlib import contextmanager

@contextmanager
def terminating(thing):
    try:
        yield thing
    finally:
        thing.terminate()

with terminating(Pool(processes=2)) as pool:
    pool.map(myFunction, mylist)
    pool.map(myfunction2, mylist2)

More Related Contents:

  1. Python multiprocessing PicklingError: Can’t pickle
  2. Can’t pickle when using multiprocessing Pool.map()
  3. Create a “with” block on several context managers? [duplicate]
  4. Explaining Python’s ‘__enter__’ and ‘__exit__’
  5. Multiprocessing: How to use Pool.map on a function defined in a class?
  6. What can multiprocessing and dill do together?
  7. Why can I pass an instance method to multiprocessing.Process, but not a multiprocessing.Pool?
  8. Understanding the Python with statement and context managers
  9. multiprocessing.Pool – PicklingError: Can’t pickle : attribute lookup thread.lock failed
  10. Alternative to contextlib.nested with variable number of context managers
  11. Catching exception in context manager __enter__()
  12. Context manager for Python’s MySQLdb
  13. How to use multiprocessing pool.map with multiple arguments
  14. What are the differences between the threading and multiprocessing modules?
  15. Combine Pool.map with shared memory Array in Python multiprocessing
  16. How do I mock an open used in a with statement (using the Mock framework in Python)?
  17. python multiprocessing on windows, if __name__ == “__main__”
  18. Sharing a complex object between processes?
  19. Filling a queue and managing multiprocessing in python
  20. What exactly is Python multiprocessing Module’s .join() Method Doing?
  21. Catch Ctrl+C / SIGINT and exit multiprocesses gracefully in python [duplicate]
  22. Python multiprocessing example not working
  23. Modify object in python multiprocessing
  24. multiprocessing pool example does not work and freeze the kernel
  25. How to use socket in Python as a context manager?
  26. Can I get a return value from multiprocessing.Process?
  27. How to utilize all cores with python multiprocessing
  28. Tensorflow and Multiprocessing: Passing Sessions
  29. How can I share a class between processes?
  30. Python multiprocessing linux windows difference

Leave a Comment