The problem is that multiprocessing must pickle things to sling them among processes, and bound methods are not picklable. The workaround (whether you consider it “easy” or not;-) is to add the infrastructure to your program to allow such methods to be pickled, registering it with the copy_reg standard library method. For example, Steven Bethard’s … Read more
Here is a list of what can be pickled. In particular, functions are only picklable if they are defined at the top-level of a module. This piece of code: import multiprocessing as mp class Foo(): @staticmethod def work(self): pass if __name__ == ‘__main__’: pool = mp.Pool() foo = Foo() pool.apply_async(foo.work) pool.close() pool.join() yields an error … Read more
is there a variant of pool.map which support multiple arguments? Python 3.3 includes pool.starmap() method: #!/usr/bin/env python3 from functools import partial from itertools import repeat from multiprocessing import Pool, freeze_support def func(a, b): return a + b def main(): a_args = [1,2,3] second_arg = 1 with Pool() as pool: L = pool.starmap(func, [(1, 1), (2, … Read more
Here are some pros/cons I came up with. Multiprocessing Pros Separate memory space Code is usually straightforward Takes advantage of multiple CPUs & cores Avoids GIL limitations for cPython Eliminates most needs for synchronization primitives unless if you use shared memory (instead, it’s more of a communication model for IPC) Child processes are interruptible/killable Python … Read more