Segmentation Fault using Python Shared Memory

Basically the problem seems to be that the underlying mmap’ed file (owned by shm within read_from_shm) is being closed when shm is garbage collected when the function returns. Then shmData refers back to it, which is where you get the segfault (for referring to a closed mmap) This seems to be a known bug, but it can be solved by keeping a reference to shm.

Additionally all SharedMemory instances want to be close()‘d with exactly one of them being unlink()‘ed when it is no longer necessary. If you don’t call shm.close() yourself, it will be called at GC as mentioned, and on Windows if it is the only one currently “open” the shared memory file will be deleted. When you call shm.close() inside store_in_shm, you introduce an OS dependency as on windows the data will be deleted, and MacOS and Linux, it will retain until unlink is called.

Finally though this doesn’t appear in your code, another problem currently exists where accessing data from independent processes (rather than child processes) can similarly delete the underlying mmap too soon. SharedMemory is a very new library, and hopefully all the kinks will work out soon.

You can re-write the given example to retain a reference to the “second” shm and just use either one to unlink:

import numpy as np
from multiprocessing import shared_memory

def store_in_shm(data):
    shm = shared_memory.SharedMemory(name="foo", create=True, size=data.nbytes)
    shmData = np.ndarray(data.shape, dtype=data.dtype, buffer=shm.buf)
    shmData[:] = data[:]
    #there must always be at least one `SharedMemory` object open for it to not
    #  be destroyed on Windows, so we won't `shm.close()` inside the function,
    #  but rather after we're done with everything.
    return shm

def read_from_shm(shape, dtype):
    shm = shared_memory.SharedMemory(name="foo", create=False)
    shmData = np.ndarray(shape, dtype, buffer=shm.buf)
    print('From read_from_shm():', shmData)
    return shm, shmData #we need to keep a reference of shm both so we don't
                        #  segfault on shmData and so we can `close()` it.

if __name__ == '__main__':
    data = np.arange(10)
    shm1 = store_in_shm(data)
    #This is where the *Windows* previously reclaimed the memory resulting in a 
    #  FileNotFoundError because the tempory mmap'ed file had been released.
    shm2, shmData = read_from_shm(data.shape, data.dtype)
    print('From __main__:', shmData)
    shm1.close() 
    shm2.close()
    #on windows "unlink" happens automatically here whether you call `unlink()` or not.
    shm2.unlink() #either shm1 or shm2