Working with big data in python and numpy, not enough ram, how to save partial results on disc?

by

Using numpy.memmap you create arrays directly mapped into a file:

import numpy
a = numpy.memmap('test.mymemmap', dtype="float32", mode="w+", shape=(200000,1000))
# here you will see a 762MB file created in your working directory

You can treat it as a conventional array:
a += 1000.

It is possible even to assign more arrays to the same file, controlling it from mutually sources if needed. But I’ve experiences some tricky things here. To open the full array you have to “close” the previous one first, using del:

del a    
b = numpy.memmap('test.mymemmap', dtype="float32", mode="r+", shape=(200000,1000))

But openning only some part of the array makes it possible to achieve the simultaneous control:

b = numpy.memmap('test.mymemmap', dtype="float32", mode="r+", shape=(2,1000))
b[1,5] = 123456.
print a[1,5]
#123456.0

Great! a was changed together with b. And the changes are already written on disk.

The other important thing worth commenting is the offset. Suppose you want to take not the first 2 lines in b, but lines 150000 and 150001.

b = numpy.memmap('test.mymemmap', dtype="float32", mode="r+", shape=(2,1000),
                 offset=150000*1000*32/8)
b[1,2] = 999999.
print a[150001,2]
#999999.0

Now you can access and update any part of the array in simultaneous operations. Note the byte-size going in the offset calculation. So for a ‘float64’ this example would be 150000*1000*64/8.

Other references:

More Related Contents:

  1. Sorting arrays in NumPy by column
  2. Computing the correlation coefficient between two multi-dimensional arrays
  3. How to normalize a NumPy array to within a certain range?
  4. How to do n-D distance and nearest neighbor calculations on numpy arrays
  5. How to extend an array in-place in Numpy?
  6. Efficient distance calculation between N points and a reference in numpy/scipy
  7. Fastest pairwise distance metric in python
  8. Numpy: Divide each row by a vector element
  9. Fitting empirical distribution to theoretical ones with Scipy (Python)?
  10. ValueError: setting an array element with a sequence
  11. Numpy `logical_or` for more than two arguments
  12. How do I convert a numpy array to (and display) an image?
  13. How does NumPy’s transpose() method permute the axes of an array?
  14. How to understand numpy strides for layman?
  15. How do I print the full NumPy array, without truncation?
  16. Multiplying across in a numpy array
  17. initialize a numpy array
  18. Use scipy.integrate.quad to integrate complex numbers
  19. What does dtype=object mean while creating a numpy array?
  20. Why is a `for` over a Python list faster than over a Numpy array?
  21. Selecting close matches from one array based on another reference array
  22. Python/Numpy MemoryError
  23. Why does numpy have a corresponding function for many ndarray methods?
  24. Efficiently sorting a numpy array in descending order?
  25. Setting the fmt option in numpy.savetxt
  26. Numpy shuffle multidimensional array by row only, keep column order unchanged
  27. Consecutive, Overlapping Subsets of Array (NumPy, Python)
  28. Computing cross-correlation function?
  29. Generate random array of floats between a range
  30. Understanding scipy deconvolve

Leave a Comment