Incremental writes to hdf5 with h5py

by

Per the FAQ, you can expand the dataset using dset.resize. For example,

import os
import h5py
import numpy as np
path="/tmp/out.h5"
os.remove(path)
with h5py.File(path, "a") as f:
    dset = f.create_dataset('voltage284', (10**5,), maxshape=(None,),
                            dtype="i8", chunks=(10**4,))
    dset[:] = np.random.random(dset.shape)        
    print(dset.shape)
    # (100000,)

    for i in range(3):
        dset.resize(dset.shape[0]+10**4, axis=0)   
        dset[-10**4:] = np.random.random(10**4)
        print(dset.shape)
        # (110000,)
        # (120000,)
        # (130000,)

More Related Contents:

  1. Optimal HDF5 dataset chunk shape for reading rows
  2. Pandas can’t read hdf5 file created with h5py
  3. How to read a v7.3 mat file via h5py?
  4. Is there an analysis speed or memory usage advantage to using HDF5 for large array storage (instead of flat binary files)?
  5. What is the difference between the two ways of accessing the hdf5 group in SVHN dataset?
  6. “Large data” workflows using pandas [closed]
  7. How to read HDF5 files in Python
  8. Saving to hdf5 is very slow (Python freezing)
  9. pytables writes much faster than h5py. Why?
  10. h5py not sticking to chunking specification?
  11. HDF5 – concurrency, compression & I/O performance [closed]
  12. which is faster for load: pickle or hdf5 in python
  13. Storing numpy sparse matrix in HDF5 (PyTables)
  14. How to get faster code than numpy.dot for matrix multiplication?
  15. Error opening file in H5PY (File signature not found)
  16. Input and output numpy arrays to h5py
  17. read matlab v7.3 file into python list of numpy arrays via h5py
  18. Pandas ParserError EOF character when reading multiple csv files to HDF5
  19. Error after upgrading pip: cannot import name ‘main’
  20. why is plotting with Matplotlib so slow?
  21. How to get most informative features for scikit-learn classifiers?
  22. Running an interactive command from within Python
  23. Display a ‘loading’ message while a time consuming function is executed in Flask
  24. Is this the way to validate Django model fields?
  25. TypeError: ‘module’ object is not callable error with driver=webdriver(“C:\\Python34\\Lib\\site-packages\\selenium\\webdriver\\chromedriver.exe”)
  26. Tkinter messagebox without window?
  27. Tensorflow: Using Adam optimizer
  28. Running Python Script as a Windows background process [duplicate]
  29. TfidfVectorizer in scikit-learn : ValueError: np.nan is an invalid document
  30. selenium webdriver upload file

Leave a Comment