If you are on Linux (or any POSIX-compliant system), you can define this array as a global variable. multiprocessing is using fork() on Linux when it starts a new child process. A newly spawned child process automatically shares the memory with its parent as long as it does not change it (copy-on-write mechanism). Since you … Read more
Update: Hey, here’s a page I just found with a compleate implmentation. Using C++/CLI, it’s quite easy to setup shared memory as per normal C++ API (C++/CLI being able to interact with the managed and native HEAP/memory references). The UnmanagedMemoryStream can then be used to expose a Stream object to C#. I did not attach … Read more
Trying again as I just saw the bounty 😉 Basically I think the error message means what it said – multiprocessing shared memory Arrays can’t be passed as arguments (by pickling). It doesn’t make sense to serialise the data – the point is the data is shared memory. So you have to make the shared … Read more
Because this is still a very high result on google and no one else has mentioned it yet, I thought I would mention the new possibility of ‘true’ shared memory which was introduced in python version 3.8.0: https://docs.python.org/3/library/multiprocessing.shared_memory.html I have here included a small contrived example (tested on linux) where numpy arrays are used, which … Read more
If you use an operating system that uses copy-on-write fork() semantics (like any common unix), then as long as you never alter your data structure it will be available to all child processes without taking up additional memory. You will not have to do anything special (except make absolutely sure you don’t alter the object). … Read more
There are two approaches: shmget and mmap. I’ll talk about mmap, since it’s more modern and flexible, but you can take a look at man shmget (or this tutorial) if you’d rather use the old-style tools. The mmap() function can be used to allocate memory buffers with highly customizable parameters to control access and permissions, … Read more