This is a little pseudo codish, but I think should be quite fast. Straightforward disk based merge, with all tables on disk. The key is that you are not doing selection per se, just indexing into the table via start/stop, which is quite fast. Selecting the rows that meet a criteria in B (using A’s … Read more
There may be a simpler way, but this is how you’d go about doing it, as far as I know: import numpy as np import tables # Generate some data x = np.random.random((100,100,100)) # Store “x” in a chunked array… f = tables.open_file(‘test.hdf’, ‘w’) atom = tables.Atom.from_dtype(x.dtype) ds = f.createCArray(f.root, ‘somename’, atom, x.shape) ds[:] = … Read more
HDF5 Advantages: Organization, flexibility, interoperability Some of the main advantages of HDF5 are its hierarchical structure (similar to folders/files), optional arbitrary metadata stored with each item, and its flexibility (e.g. compression). This organizational structure and metadata storage may sound trivial, but it’s very useful in practice. Another advantage of HDF is that the datasets can … Read more
Heres a complete example. import numpy as np import pandas as pd import os fname=”groupby.h5″ # create a frame df = pd.DataFrame({‘A’: [‘foo’, ‘foo’, ‘foo’, ‘foo’, ‘bar’, ‘bar’, ‘bar’, ‘bar’, ‘foo’, ‘foo’, ‘foo’], ‘B’: [‘one’, ‘one’, ‘one’, ‘two’, ‘one’, ‘one’, ‘one’, ‘two’, ‘two’, ‘two’, ‘one’], ‘C’: [‘dull’, ‘dull’, ‘shiny’, ‘dull’, ‘dull’, ‘shiny’, ‘shiny’, ‘dull’, ‘shiny’, … Read more
np.dot dispatches to BLAS when NumPy has been compiled to use BLAS, a BLAS implementation is available at run-time, your data has one of the dtypes float32, float64, complex32 or complex64, and the data is suitably aligned in memory. Otherwise, it defaults to using its own, slow, matrix multiplication routine. Checking your BLAS linkage is … Read more
The answer by DaveP is almost right… but can cause problems for very sparse matrices: if the last column(s) or row(s) are empty, they are dropped. So to be sure that everything works, the “shape” attribute must be stored too. This is the code I regularly use: import tables as tb from numpy import array … Read more
This is an interesting comparison of PyTables and h5py write performance. Typically I use them to read HDF5 files (and usually with a few reads of large datasets), so haven’t noticed this difference. My thoughts align with @max9111: that performance should improve as the number of write operations decreased as the size of the written … Read more
These examples show how to use h5py to copy datasets between 2 HDF5 files. See my other answer for PyTables examples. I created some simple HDF5 files to mimic CSV type data (all floats, but the process is the same if you have mixed data types). Based on your description, each file only has one … Read more