How do I pass large numpy arrays between python subprocesses without saving to disk?

by

While googling around for more information about the code Joe Kington posted, I found the numpy-sharedmem package. Judging from this numpy/multiprocessing tutorial it seems to share the same intellectual heritage (maybe largely the same authors? — I’m not sure).

Using the sharedmem module, you can create a shared-memory numpy array (awesome!), and use it with multiprocessing like this:

import sharedmem as shm
import numpy as np
import multiprocessing as mp

def worker(q,arr):
    done = False
    while not done:
        cmd = q.get()
        if cmd == 'done':
            done = True
        elif cmd == 'data':
            ##Fake data. In real life, get data from hardware.
            rnd=np.random.randint(100)
            print('rnd={0}'.format(rnd))
            arr[:]=rnd
        q.task_done()

if __name__=='__main__':
    N=10
    arr=shm.zeros(N,dtype=np.uint8)
    q=mp.JoinableQueue()    
    proc = mp.Process(target=worker, args=[q,arr])
    proc.daemon=True
    proc.start()

    for i in range(3):
        q.put('data')
        # Wait for the computation to finish
        q.join()   
        print arr.shape
        print(arr)
    q.put('done')
    proc.join()

Running yields

rnd=53
(10,)
[53 53 53 53 53 53 53 53 53 53]
rnd=15
(10,)
[15 15 15 15 15 15 15 15 15 15]
rnd=87
(10,)
[87 87 87 87 87 87 87 87 87 87]

More Related Contents:

  1. Getting data from ctypes array into numpy
  2. Passing Numpy arrays to a C function for input and output
  3. How to use NumPy array with ctypes?
  4. Binning a column with Python Pandas
  5. Fitting empirical distribution to theoretical ones with Scipy (Python)?
  6. Simple Digit Recognition OCR in OpenCV-Python
  7. Best way to join / merge by range in pandas
  8. How do you use the ellipsis slicing syntax in Python?
  9. How do I pipe a subprocess call to a text file?
  10. Concatenate two NumPy arrays vertically
  11. How to make scipy.interpolate give an extrapolated result beyond the input range?
  12. Inverse Distance Weighted (IDW) Interpolation with Python
  13. Convert NumPy array to Python list
  14. Display realtime output of a subprocess in a tkinter widget
  15. Relationship between SciPy and NumPy
  16. Multivariate spline interpolation in python/scipy?
  17. How to conditionally combine two numpy arrays of the same shape
  18. scipy curve_fit doesn’t like math module
  19. Numpy Pure Functions for performance, caching
  20. Pythonic way of detecting outliers in one dimensional observation data
  21. RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  22. How do I convert an OpenCV image (BGR and BGRA) to a pygame.Surface object
  23. Why are frequency values rounded in signal using FFT?
  24. Is there a numpy/scipy dot product, calculating only the diagonal entries of the result?
  25. How can I draw lines into numpy arrays?
  26. subprocess call ffmpeg (command line)
  27. How can I capture the stdout output of a child process?
  28. Get the indices of N highest values in an ndarray
  29. Interweaving two NumPy arrays efficiently
  30. Numpy.dtype has the wrong size, try recompiling

Leave a Comment