TimeDistributed(Dense) vs Dense in Keras – Same number of parameters

by

TimeDistributedDense applies a same dense to every time step during GRU/LSTM Cell unrolling. So the error function will be between predicted label sequence and the actual label sequence. (Which is normally the requirement for sequence to sequence labeling problems).

However, with return_sequences=False, Dense layer is applied only once at the last cell. This is normally the case when RNNs are used for classification problem. If return_sequences=True then Dense layer is applied to every timestep just like TimeDistributedDense.

So for as per your models both are same, but if you change your second model to return_sequences=False, then Dense will be applied only at the last cell. Try changing it and the model will throw as error because then the Y will be of size [Batch_size, InputSize], it is no more a sequence to sequence but a full sequence to label problem.

from keras.models import Sequential
from keras.layers import Dense, Activation, TimeDistributed
from keras.layers.recurrent import GRU
import numpy as np

InputSize = 15
MaxLen = 64
HiddenSize = 16

OutputSize = 8
n_samples = 1000

model1 = Sequential()
model1.add(GRU(HiddenSize, return_sequences=True, input_shape=(MaxLen, InputSize)))
model1.add(TimeDistributed(Dense(OutputSize)))
model1.add(Activation('softmax'))
model1.compile(loss="categorical_crossentropy", optimizer="rmsprop")


model2 = Sequential()
model2.add(GRU(HiddenSize, return_sequences=True, input_shape=(MaxLen, InputSize)))
model2.add(Dense(OutputSize))
model2.add(Activation('softmax'))
model2.compile(loss="categorical_crossentropy", optimizer="rmsprop")

model3 = Sequential()
model3.add(GRU(HiddenSize, return_sequences=False, input_shape=(MaxLen, InputSize)))
model3.add(Dense(OutputSize))
model3.add(Activation('softmax'))
model3.compile(loss="categorical_crossentropy", optimizer="rmsprop")

X = np.random.random([n_samples,MaxLen,InputSize])
Y1 = np.random.random([n_samples,MaxLen,OutputSize])
Y2 = np.random.random([n_samples, OutputSize])

model1.fit(X, Y1, batch_size=128, nb_epoch=1)
model2.fit(X, Y1, batch_size=128, nb_epoch=1)
model3.fit(X, Y2, batch_size=128, nb_epoch=1)

print(model1.summary())
print(model2.summary())
print(model3.summary())

In the above example architecture of model1 and model2 are sample (sequence to sequence models) and model3 is a full sequence to label model.

More Related Contents:

  1. Many to one and many to many LSTM examples in Keras
  2. Why binary_crossentropy and categorical_crossentropy give different performances for the same problem?
  3. Loss & accuracy – Are these reasonable learning curves?
  4. How to calculate optimal batch size
  5. What is the difference between loss function and metric in Keras? [duplicate]
  6. Keras Text Preprocessing – Saving Tokenizer object to file for scoring
  7. Is deep learning bad at fitting simple non linear functions outside training scope (extrapolating)?
  8. What is the difference between Keras model.evaluate() and model.predict()?
  9. Using pre-trained word2vec with LSTM for word generation
  10. Keras give input to intermediate layer and get final output
  11. Is this possible to predict the lottery numbers (not the most accurate)? [closed]
  12. Keras Dense layer’s input is not flattened
  13. Clarification on a Neural Net that plays Snake
  14. Common causes of nans during training of neural networks
  15. Keras Sequential model input layer
  16. How to concatenate two layers in keras?
  17. How to interpret MSE in Keras Regressor
  18. Epoch vs Iteration when training neural networks [closed]
  19. What is `weight_decay` meta parameter in Caffe?
  20. Unable to approximate the sine function using a neural network
  21. Why should weights of Neural Networks be initialized to random numbers? [closed]
  22. multi-layer perceptron (MLP) architecture: criteria for choosing number of hidden layers and size of the hidden layer? [closed]
  23. What is the role of TimeDistributed layer in Keras?
  24. Caffe: What can I do if only a small batch fits into memory?
  25. How to reduce a fully-connected (`”InnerProduct”`) layer using truncated SVD
  26. keras: how to save the training history attribute of the history object
  27. How to add and remove new layers in keras after loading weights?
  28. Can I send callbacks to a KerasClassifier?
  29. How can I get biases from a trained model in Keras?
  30. Choosing number of Steps per Epoch

Leave a Comment