Because this is still a very high result on google and no one else has mentioned it yet, I thought I would mention the new possibility of ‘true’ shared memory which was introduced in python version 3.8.0: https://docs.python.org/3/library/multiprocessing.shared_memory.html I have here included a small contrived example (tested on linux) where numpy arrays are used, which … Read more
What Giulio Franco says is true for multithreading vs. multiprocessing in general. However, Python* has an added issue: There’s a Global Interpreter Lock that prevents two threads in the same process from running Python code at the same time. This means that if you have 8 cores, and change your code to use 8 threads, … Read more
After some more googling I found the answer here. It turns out that certain Python modules (numpy, scipy, tables, pandas, skimage…) mess with core affinity on import. As far as I can tell, this problem seems to be specifically caused by them linking against multithreaded OpenBLAS libraries. A workaround is to reset the task affinity … Read more
If you use an operating system that uses copy-on-write fork() semantics (like any common unix), then as long as you never alter your data structure it will be available to all child processes without taking up additional memory. You will not have to do anything special (except make absolutely sure you don’t alter the object). … Read more
Back in the old days of Python, to call a function with arbitrary arguments, you would use apply: apply(f,args,kwargs) apply still exists in Python2.7 though not in Python3, and is generally not used anymore. Nowadays, f(*args,**kwargs) is preferred. The multiprocessing.Pool modules tries to provide a similar interface. Pool.apply is like Python apply, except that the … Read more
When you use multiprocessing to open a second process, an entirely new instance of Python, with its own global state, is created. That global state is not shared, so changes made by child processes to global variables will be invisible to the parent process. Additionally, most of the abstractions that multiprocessing provides use pickle to … Read more
A general answer involves using a Manager object. Adapted from the docs: from multiprocessing import Process, Manager def f(d): d[1] += ‘1’ d[‘2’] += 2 if __name__ == ‘__main__’: manager = Manager() d = manager.dict() d[1] = ‘1’ d[‘2’] = 2 p1 = Process(target=f, args=(d,)) p2 = Process(target=f, args=(d,)) p1.start() p2.start() p1.join() p2.join() print d … Read more
You could use threading or multiprocessing. Due to peculiarities of CPython, threading is unlikely to achieve true parallelism. For this reason, multiprocessing is generally a better bet. Here is a complete example: from multiprocessing import Process def func1(): print ‘func1: starting’ for i in xrange(10000000): pass print ‘func1: finishing’ def func2(): print ‘func2: starting’ for … Read more
On Windows the subprocesses will import (i.e. execute) the main module at start. You need to insert an if __name__ == ‘__main__’: guard in the main module to avoid creating subprocesses recursively. Modified testMain.py: import parallelTestModule if __name__ == ‘__main__’: extractor = parallelTestModule.ParallelExtractor() extractor.runInParallel(numProcesses=2, numThreads=4)
To add to @unutbu’s (not available anymore) and @Henry Gomersall’s answers. You could use shared_arr.get_lock() to synchronize access when needed: shared_arr = mp.Array(ctypes.c_double, N) # … def f(i): # could be anything numpy accepts as an index such another numpy array with shared_arr.get_lock(): # synchronize access arr = np.frombuffer(shared_arr.get_obj()) # no data copying arr[i] = … Read more