Fastest save and load options for a numpy array

by

For really big arrays, I’ve heard about several solutions, and they mostly on being lazy on the I/O :

  • NumPy.memmap, maps big arrays to binary form
    • Pros :
      • No dependency other than Numpy
      • Transparent replacement of ndarray (Any class accepting ndarray accepts memmap )
    • Cons :
      • Chunks of your array are limited to 2.5G
      • Still limited by Numpy throughput

  • Use Python bindings for HDF5, a bigdata-ready file format, like PyTables or h5py

    • Pros :
      • Format supports compression, indexing, and other super nice features
      • Apparently the ultimate PetaByte-large file format
    • Cons :
      • Learning curve of having a hierarchical format ?
      • Have to define what your performance needs are (see later)

  • Python’s pickling system (out of the race, mentioned for Pythonicity rather than speed)

    • Pros:
      • It’s Pythonic ! (haha)
      • Supports all sorts of objects
    • Cons:
      • Probably slower than others (because aimed at any objects not arrays)

Numpy.memmap

From the docs of NumPy.memmap :

Create a memory-map to an array stored in a binary file on disk.

Memory-mapped files are used for accessing small segments of large files on disk, without reading the entire file into memory

The memmap object can be used anywhere an ndarray is accepted. Given any memmap fp , isinstance(fp, numpy.ndarray) returns True.

HDF5 arrays

From the h5py doc

Lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want.

The format supports compression of data in various ways (more bits loaded for same I/O read), but this means that the data becomes less easy to query individually, but in your case (purely loading / dumping arrays) it might be efficient

More Related Contents:

  1. Performance of Pandas apply vs np.vectorize to create new column from existing columns
  2. numpy: most efficient frequency counts for unique values in an array
  3. Why is numpy’s einsum faster than numpy’s built in functions?
  4. Numpy sum elements in array based on its value
  5. Efficiently return the index of the first value satisfying condition in array
  6. Frequency counts for unique values in a NumPy array
  7. Most efficient way to forward-fill NaN values in numpy array
  8. Numpy: Fix array with rows of different lengths by filling the empty elements with zeros
  9. Why is a `for` over a Python list faster than over a Numpy array?
  10. Why is Numpy much faster at creating a Zero array compared to replacing the values of an existing array with zeros?
  11. Efficient dot products of large memory-mapped arrays
  12. Creating a numpy array of 3D coordinates from three 1D arrays
  13. Sorting arrays in NumPy by column
  14. Python/NumPy first occurrence of subarray
  15. Numpy matrix to array
  16. Replacing Numpy elements if condition is met
  17. How to make a multidimension numpy array with a varying row size?
  18. Implement MATLAB’s im2col ‘sliding’ in Python
  19. How to normalize a NumPy array to within a certain range?
  20. subsampling every nth entry in a numpy array
  21. Python’s sum vs. NumPy’s numpy.sum
  22. Numpy and line intersections
  23. Numpy: Creating a complex array from 2 real ones?
  24. Python code performance decreases with threading
  25. Why list comprehension is much faster than numpy for multiplying arrays?
  26. Repeat each values of an array different times
  27. How to create 2d array with numpy random.choice for every rows?
  28. What is the difference between np.array() and np.asarray()?
  29. Partition array into N chunks with Numpy
  30. Convert list of lists with different lengths to a numpy array [duplicate]

Leave a Comment