Why do we “pack” the sequences in PyTorch?

by

I have stumbled upon this problem too and below is what I figured out.

When training RNN (LSTM or GRU or vanilla-RNN), it is difficult to batch the variable length sequences. For example: if the length of sequences in a size 8 batch is [4,6,8,5,4,3,7,8], you will pad all the sequences and that will result in 8 sequences of length 8. You would end up doing 64 computations (8×8), but you needed to do only 45 computations. Moreover, if you wanted to do something fancy like using a bidirectional-RNN, it would be harder to do batch computations just by padding and you might end up doing more computations than required.

Instead, PyTorch allows us to pack the sequence, internally packed sequence is a tuple of two lists. One contains the elements of sequences. Elements are interleaved by time steps (see example below) and other contains the size of each sequence the batch size at each step. This is helpful in recovering the actual sequences as well as telling RNN what is the batch size at each time step. This has been pointed by @Aerin. This can be passed to RNN and it will internally optimize the computations.

I might have been unclear at some points, so let me know and I can add more explanations.

Here’s a code example:

 a = [torch.tensor([1,2,3]), torch.tensor([3,4])]
 b = torch.nn.utils.rnn.pad_sequence(a, batch_first=True)
 >>>>
 tensor([[ 1,  2,  3],
    [ 3,  4,  0]])
 torch.nn.utils.rnn.pack_padded_sequence(b, batch_first=True, lengths=[3,2])
 >>>>PackedSequence(data=tensor([ 1,  3,  2,  4,  3]), batch_sizes=tensor([ 2,  2,  1]))

More Related Contents:

  1. What’s the difference between “hidden” and “output” in PyTorch LSTM?
  2. Best way to save a trained model in PyTorch? [closed]
  3. How do I save a trained model in PyTorch?
  4. Pytorch – RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed
  5. Keras input explanation: input_shape, units, batch_size, dim, etc
  6. Keras : How should I prepare input data for RNN?
  7. Many to one and many to many LSTM examples in Keras
  8. How to initialize weights in PyTorch?
  9. How do I initialize weights in PyTorch?
  10. Why do we need to call zero_grad() in PyTorch?
  11. How does Pytorch’s “Fold” and “Unfold” work?
  12. What does model.eval() do in pytorch?
  13. how to format the image data for training/prediction when images are different in size?
  14. PyTorch NotImplementedError in forward
  15. How to understand the term `tensor` in TensorFlow?
  16. Calculate the output size in convolution layer [closed]
  17. LSTM module for Caffe
  18. How to speed up Gensim Word2vec model load time?
  19. What’s the difference between torch.stack() and torch.cat() functions?
  20. Error when checking model input: expected convolution2d_input_1 to have 4 dimensions, but got array with shape (32, 32, 3)
  21. Is .data still useful in pytorch?
  22. ValueError: Tensor must be from the same graph as Tensor with Bidirectinal RNN in Tensorflow
  23. RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same
  24. How do I split a custom dataset into training and test datasets?
  25. RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! when resuming training
  26. PyTorch Binary Classification – same network structure, ‘simpler’ data, but worse performance?
  27. What is `weight_decay` meta parameter in Caffe?
  28. How to avoid: read.table truncates numeric values beginning with 0
  29. ModuleNotFoundError: No module named ‘tools.nnwrap’
  30. How to fix “ResourceExhaustedError: OOM when allocating tensor”

Leave a Comment