UPDATE: nowadays I would choose between Parquet, Feather (Apache Arrow), HDF5 and Pickle. Pro’s and Contra’s: Parquet pros one of the fastest and widely supported binary storage formats supports very fast compression methods (for example Snappy codec) de-facto standard storage format for Data Lakes / BigData contras the whole dataset must be read into memory. … Read more
Updated to use pandas 0.13.1 1) No. http://pandas.pydata.org/pandas-docs/dev/io.html#notes-caveats. There are various ways to do this, e.g. have your different threads/processes write out the computation results, then have a single process combine. 2) depending the type of data you store, how you do it, and how you want to retrieve, HDF5 can offer vastly better performance. … Read more
The problem It seems like there is indeed a conflict with the order of elements in arrays: matlab arranges the elements from the first dimension to the last (like fortran), while caffe and hdf5 stores the arrays from last dimension to first: Suppose we have X of shape nxcxhxw then the “second element of X” … Read more
These examples show how to use h5py to copy datasets between 2 HDF5 files. See my other answer for PyTables examples. I created some simple HDF5 files to mimic CSV type data (all floats, but the process is the same if you have mixed data types). Based on your description, each file only has one … Read more
Writing Data to HDF5 If you write to a chunked dataset without specifying a chunkshape, h5py will do that automaticly for you. Since h5py can’t know how do you wan’t to write or read the data from the dataset, this will often end up in a bad performance. You also use the default chunk-cache-size of … Read more
Read HDF5 import h5py filename = “file.hdf5” with h5py.File(filename, “r”) as f: # Print all root level object names (aka keys) # these can be group or dataset names print(“Keys: %s” % f.keys()) # get first object name/key; may or may NOT be a group a_group_key = list(f.keys())[0] # get the object type for a_group_key: … Read more
Finding the right chunk cache size At first I want to discuss some general things. It is very important to know that each individual chunk could only be read or written as a whole. The standard chunk-cache size of h5py which can avoid excessive disk I/Os is only one MB per default and should in … Read more
I routinely use tens of gigabytes of data in just this fashion e.g. I have tables on disk that I read via queries, create data and append back. It’s worth reading the docs and late in this thread for several suggestions for how to store your data. Details which will affect how you store your … Read more