[caffe]: check fails: Check failed: hdf_blobs_[i]->shape(0) == num (200 vs. 6000)

by

The problem

It seems like there is indeed a conflict with the order of elements in arrays: matlab arranges the elements from the first dimension to the last (like fortran), while caffe and hdf5 stores the arrays from last dimension to first:
Suppose we have X of shape nxcxhxw then the “second element of X” is X[2,1,1,1] in matlab but X[0,0,0,1] in C (1-based vs 0-based indexing doesn’t make life easier at all).
Therefore, when you save an array of size=[200, 6000, 1, 1] in Matlab, what hdf5 and caffe are actually seeing is as array of shape=[6000,200].

Using the h5ls command line tool can help you spot the problem.
In matlab you saved

>> hdf5write('my_data.h5', '/new_train_x', 
  single( reshape(new_train_x,[200, 6000, 1, 1]) );
>> hdf5write('my_data.h5', '/label_train', 
  single( reshape(label_train,[200, 1, 1, 1]) ),
  'WriteMode', 'append' );

Now you can inspect the resulting my_data.h5 using h5ls (in Linux terminal):

user@host:~/$ h5ls ./my_data.h5
  label_train              Dataset {200}
  new_train_x              Dataset {6000, 200}

As you can see, the arrays are written “backwards”.

Solution

Taking this conflict into account when exporting data from Matlab, you should permute:

load data.mat
hdf5write('my_data.h5', '/new_train_x', 
  single( permute(reshape(new_train_x,[200, 6000, 1, 1]),[4:-1:1] ) );
hdf5write('my_data.h5', '/label_train', 
  single( permute(reshape(label_train,[200, 1, 1, 1]), [4:-1:1] ) ),
  'WriteMode', 'append' );

Inspect the resulting my_data.h5 using h5ls now results with:

user@host:~/$ h5ls ./my_data.h5
  label_train              Dataset {200, 1, 1, 1}
  new_train_x              Dataset {200, 6000, 1, 1}

Which is what you expected in the first place.

More Related Contents:

  1. How to training/testing my own dataset in caffe?
  2. Is there any standard rule for considering the best result or at which stage I have to stop train the data with minimum error
  3. Test labels for regression caffe, float not allowed?
  4. Common causes of nans during training of neural networks
  5. What is a `”Python”` layer in caffe?
  6. InfogainLoss layer
  7. How to interpret caffe log with debug_info?
  8. Caffe | solver.prototxt values setting strategy
  9. What is `weight_decay` meta parameter in Caffe?
  10. Neural network in MATLAB
  11. How do I load a caffe model and convert to a numpy array?
  12. How to feed caffe multi label data in HDF5 format?
  13. LSTM module for Caffe
  14. caffe data layer example step by step
  15. Does Caffe need data to be shuffled?
  16. Tackling Class Imbalance: scaling contribution to loss and sgd
  17. MATLAB: Differences between .mat versions
  18. Caffe: What can I do if only a small batch fits into memory?
  19. scale the loss value according to “badness” in caffe
  20. How to create caffe.deploy from train.prototxt
  21. How to reduce a fully-connected (`”InnerProduct”`) layer using truncated SVD
  22. What is `lr_policy` in Caffe?
  23. caffe: model definition: write same layer with different phase using caffe.NetSpec()
  24. Understanding Matlab FFT example
  25. Nearest-neighbor interpolation algorithm in MATLAB
  26. character reconstruction and filling for OCR
  27. How to create an executable .exe file from a .m file
  28. Analytical Fourier transform vs FFT of functions in Matlab
  29. How to deal with overflow and underflow?
  30. Multiple outputs in Keras

Leave a Comment