First, there is a minor difference in output from your 2 methods. Method 1: returns the full array (of the encoded file name) Method 2: only returns the first element (character) of the array Let’s deconstruct your code to understand what you have. The first part deals with h5py data objects. f[‘digitStruct’] -> returns a … Read more
Version 7.3 of MAT-files uses HDF5 format, this format has a significant storage overhead to describe the contents of the file, especially so for complex nested cellarrays and structures. Its main advantage over previous versions of MAT-files is that it allows storing data larger than 2GB on 64-bit systems. Note that both v7 and v7.3 … Read more
I had a similar problem. The line listed with the ‘EOF inside string’ had a string that contained within it a single quote mark (‘). When I added the option quoting=csv.QUOTE_NONE it fixed my problem. For example: import csv df = pd.read_csv(csvfile, header = None, delimiter=”\t”, quoting=csv.QUOTE_NONE, encoding=’utf-8’)
HDF5 Advantages: Organization, flexibility, interoperability Some of the main advantages of HDF5 are its hierarchical structure (similar to folders/files), optional arbitrary metadata stored with each item, and its flexibility (e.g. compression). This organizational structure and metadata storage may sound trivial, but it’s very useful in practice. Another advantage of HDF is that the datasets can … Read more
Matlab 7.3 file format is not extremely easy to work with h5py. It relies on HDF5 reference, cf. h5py documentation on references. >>> import h5py >>> f = h5py.File(‘test.mat’) >>> list(f.keys()) [‘#refs#’, ‘struArray’] >>> struArray = f[‘struArray’] >>> struArray[‘name’][0, 0] # this is the HDF5 reference <HDF5 object reference> >>> f[struArray[‘name’][0, 0]].value # this is … Read more
I’ve worked a little on the pytables module in pandas.io and from what I know pandas interaction with HDF files is limited to specific structures that pandas understands. To see what these look like, you can try import pandas as pd import numpy as np pd.Series(np.zeros((3,5),dtype=np.float32).to_hdf(‘test.h5′,’test’) If you open ‘test.h5’ in HDFView, you will see … Read more
np.dot dispatches to BLAS when NumPy has been compiled to use BLAS, a BLAS implementation is available at run-time, your data has one of the dtypes float32, float64, complex32 or complex64, and the data is suitably aligned in memory. Otherwise, it defaults to using its own, slow, matrix multiplication routine. Checking your BLAS linkage is … Read more
The answer by DaveP is almost right… but can cause problems for very sparse matrices: if the last column(s) or row(s) are empty, they are dropped. So to be sure that everything works, the “shape” attribute must be stored too. This is the code I regularly use: import tables as tb from numpy import array … Read more
Per the FAQ, you can expand the dataset using dset.resize. For example, import os import h5py import numpy as np path=”/tmp/out.h5″ os.remove(path) with h5py.File(path, “a”) as f: dset = f.create_dataset(‘voltage284’, (10**5,), maxshape=(None,), dtype=”i8″, chunks=(10**4,)) dset[:] = np.random.random(dset.shape) print(dset.shape) # (100000,) for i in range(3): dset.resize(dset.shape[0]+10**4, axis=0) dset[-10**4:] = np.random.random(10**4) print(dset.shape) # (110000,) # (120000,) # … Read more