How to training/testing my own dataset in caffe?

by

I think the most straight forward way to transfer data from Matlab to caffe is via HDF5 file.

First, save your data in Matlab in an HDF5 file using hdf5write. I assume your training data is stored in a variable name X of size 300-by-30 and the labels are stored in y a 300-by-1 vector:

hdf5write('my_data.h5', '/X', 
  single( permute(reshape(X,[300, 30, 1, 1]),[4:-1:1]) ) );
hdf5write('my_data.h5', '/label', 
  single( permute(reshape(y,[300, 1, 1, 1]),[4:-1:1]) ), 
  'WriteMode', 'append' );

Note that the data is saved as a 4D array: the first dimension is the number of features, second one is the feature’s dimension and the last two are 1 (representing no spatial dimensions). Also note that the names given to the data in the HDF5 are "X" and "label" – these names should be used as the "top" blobs of the input data layer.

Why permute? please see this answer for an explanation.

You also need to prepare a text file listing the names of all hdf5 files you are using (in your case, only my_data.h5). File /path/to/list/file.txt should have a single line

/path/to/my_data.h5

Now you can add an input data layer to your train_val.prototxt

layer {
  type: "HDF5Data"
  name: "data"
  top: "X"     # note: same name as in HDF5
  top: "label" # 
  hdf5_data_param {
    source: "/path/to/list/file.txt"
    batch_size: 20
  }
  include { phase: TRAIN }
}

For more information regarding hdf5 input layer, you can see in this answer.

More Related Contents:

  1. [caffe]: check fails: Check failed: hdf_blobs_[i]->shape(0) == num (200 vs. 6000)
  2. Is there any standard rule for considering the best result or at which stage I have to stop train the data with minimum error
  3. Test labels for regression caffe, float not allowed?
  4. Common causes of nans during training of neural networks
  5. What is a `”Python”` layer in caffe?
  6. InfogainLoss layer
  7. How to interpret caffe log with debug_info?
  8. Caffe | solver.prototxt values setting strategy
  9. What is `weight_decay` meta parameter in Caffe?
  10. Neural network in MATLAB
  11. How do I load a caffe model and convert to a numpy array?
  12. How to feed caffe multi label data in HDF5 format?
  13. LSTM module for Caffe
  14. caffe data layer example step by step
  15. Does Caffe need data to be shuffled?
  16. Tackling Class Imbalance: scaling contribution to loss and sgd
  17. Caffe: What can I do if only a small batch fits into memory?
  18. scale the loss value according to “badness” in caffe
  19. How to create caffe.deploy from train.prototxt
  20. How to reduce a fully-connected (`”InnerProduct”`) layer using truncated SVD
  21. What is `lr_policy` in Caffe?
  22. caffe: model definition: write same layer with different phase using caffe.NetSpec()
  23. How do you concatenate the rows of a matrix into a vector?
  24. Sort a matrix with another matrix
  25. Get the indices of the n largest elements in a matrix
  26. Matlab: implementing what CTRL+C does, but in the code
  27. How can I divide each row of a matrix by a fixed row?
  28. Octave/Matlab: Adding new elements to a vector
  29. Solving an ODE when the function is given as discrete values -matlab-
  30. How to nest multiple parfor loops

Leave a Comment