save numpy array in append mode

by

The build-in .npy file format is perfectly fine for working with small datasets, without relying on external modules other then numpy.

However, when you start having large amounts of data, the use of a file format, such as HDF5, designed to handle such datasets, is to be preferred [1].

For instance, below is a solution to save numpy arrays in HDF5 with PyTables,

Step 1: Create an extendable EArray storage 

import tables
import numpy as np

filename="outarray.h5"
ROW_SIZE = 100
NUM_COLUMNS = 200

f = tables.open_file(filename, mode="w")
atom = tables.Float64Atom()

array_c = f.create_earray(f.root, 'data', atom, (0, ROW_SIZE))

for idx in range(NUM_COLUMNS):
    x = np.random.rand(1, ROW_SIZE)
    array_c.append(x)
f.close()

Step 2: Append rows to an existing dataset (if needed)

f = tables.open_file(filename, mode="a")
f.root.data.append(x)

Step 3: Read back a subset of the data

f = tables.open_file(filename, mode="r")
print(f.root.data[1:10,2:20]) # e.g. read from disk only this part of the dataset

More Related Contents:

  1. Setting the fmt option in numpy.savetxt
  2. Efficiently Using Multiple Numpy Slices for Random Image Cropping
  3. What is the difference between flatten and ravel functions in numpy?
  4. Computing the correlation coefficient between two multi-dimensional arrays
  5. How to return 0 with divide by zero
  6. Problems with pip install numpy – RuntimeError: Broken toolchain: cannot link a simple C program
  7. How do I fit a sine curve to my data with pylab and numpy?
  8. Most efficient way to find mode in numpy array
  9. numpy: formal definition of “array_like” objects?
  10. Most efficient way to forward-fill NaN values in numpy array
  11. Calculating the area under a curve given a set of coordinates, without knowing the function
  12. Why does numpy.zeros takes up little space
  13. Generalise slicing operation in a NumPy array
  14. how is axis indexed in numpy’s array?
  15. Efficiently Calculating a Euclidean Distance Matrix Using Numpy
  16. Convert RGB to black OR white
  17. Map each list value to its corresponding percentile
  18. “Converting” Numpy arrays to Matlab and vice versa
  19. Most efficient way to calculate radial profile
  20. Adding a field to a structured numpy array
  21. NumPy array/matrix of mixed types
  22. How can I upgrade NumPy?
  23. Plotting implicit equations in 3d
  24. How to resample a dataframe with different functions applied to each column?
  25. Converting 2D Numpy array of grayscale values to a PIL image
  26. Convert a column of timestamps into periods in pandas
  27. Consecutive, Overlapping Subsets of Array (NumPy, Python)
  28. Why can’t I raise to a negative power in numpy?
  29. Generate random array of floats between a range
  30. Understanding scipy deconvolve

Leave a Comment