Can’t pickle when using multiprocessing Pool.map()

by

The problem is that multiprocessing must pickle things to sling them among processes, and bound methods are not picklable. The workaround (whether you consider it “easy” or not;-) is to add the infrastructure to your program to allow such methods to be pickled, registering it with the copy_reg standard library method.

For example, Steven Bethard’s contribution to this thread (towards the end of the thread) shows one perfectly workable approach to allow method pickling/unpickling via copy_reg.

More Related Contents:

  1. Multiprocessing vs Threading Python [duplicate]
  2. Python multiprocessing PicklingError: Can’t pickle
  3. How to run functions in parallel?
  4. multiprocessing.Pool: When to use apply, apply_async or map?
  5. What are the differences between the threading and multiprocessing modules?
  6. Keyboard Interrupts with python’s multiprocessing Pool
  7. Python Process Pool non-daemonic?
  8. Combine Pool.map with shared memory Array in Python multiprocessing
  9. Multiprocessing: How to use Pool.map on a function defined in a class?
  10. How to use multiprocessing queue in Python?
  11. Unable to load files using pickle and multiple modules
  12. What can multiprocessing and dill do together?
  13. Processing single file from multiple processes
  14. multiprocessing vs multithreading vs asyncio in Python 3
  15. Filling a queue and managing multiprocessing in python
  16. Multiprocessing.Pool makes Numpy matrix multiplication slower
  17. Can I use a multiprocessing Queue in a function called by Pool.imap?
  18. multiprocessing.Pool() slower than just using ordinary functions
  19. Why can I pass an instance method to multiprocessing.Process, but not a multiprocessing.Pool?
  20. How do you pass a Queue reference to a function managed by pool.map_async()?
  21. Passing multiple parameters to pool.map() function in Python [duplicate]
  22. What kind of problems (if any) would there be combining asyncio with multiprocessing?
  23. Python Multiprocessing Lib Error (AttributeError: __exit__)
  24. multiprocessing vs multithreading vs asyncio
  25. Python Multiprocess Pool. How to exit the script when one of the worker process determines no more work needs to be done?
  26. How to let Pool.map take a lambda function
  27. Python: Writing to a single file with queue while using multiprocessing Pool
  28. Python multiprocessing – tracking the process of pool.map operation
  29. Using multiprocessing.Process with a maximum number of simultaneous processes
  30. How to use Python multiprocessing Pool.map to fill numpy array in a for loop

Leave a Comment