How to add an attention mechanism in keras?

by

If you want to have an attention along the time dimension, then this part of your code seems correct to me:

activations = LSTM(units, return_sequences=True)(embedded)

# compute importance for each step
attention = Dense(1, activation='tanh')(activations)
attention = Flatten()(attention)
attention = Activation('softmax')(attention)
attention = RepeatVector(units)(attention)
attention = Permute([2, 1])(attention)

sent_representation = merge([activations, attention], mode="mul")

You’ve worked out the attention vector of shape (batch_size, max_length):

attention = Activation('softmax')(attention)

I’ve never seen this code before, so I can’t say if this one is actually correct or not:

K.sum(xin, axis=-2)

Further reading (you might have a look):

More Related Contents:

  1. Keras Dense layer’s input is not flattened
  2. How to get reproducible results in keras
  3. Tensorflow – ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type float)
  4. Getting gradient of model output w.r.t weights using Keras
  5. Proper way to feed time-series data to stateful LSTM?
  6. How to use a Keras RNN model to forecast for future dates or events?
  7. TypeError: __array__() takes 1 positional argument but 2 were given (Image classification Keras)
  8. Tensorflow 2.0 – AttributeError: module ‘tensorflow’ has no attribute ‘Session’
  9. Keras – Plot training, validation and test set accuracy
  10. Neural Network LSTM input shape from dataframe
  11. Passing output of a CNN to BILSTM
  12. Get class labels from Keras functional model
  13. How do I check if keras is using gpu version of tensorflow?
  14. How does Keras handle multilabel classification?
  15. Keras: .predict returns percentages instead of classes
  16. Keras Custom loss function to pass arguments other than y_true and y_pred
  17. Save and load weights in keras
  18. Unknown initializer: GlorotUniform when loading Keras model
  19. Tensorflow : logits and labels must have the same first dimension
  20. How to add and remove new layers in keras after loading weights?
  21. How to log Keras loss output to a file
  22. How to fix “AttributeError: module ‘tensorflow’ has no attribute ‘get_default_graph'”?
  23. ‘Tensor’ object has no attribute ‘lower’
  24. ValueError: Output tensors to a Model must be the output of a TensorFlow `Layer`
  25. How do I plot a Keras/Tensorflow subclassing API model?
  26. Strange behaviour of the loss function in keras model, with pretrained convolutional base
  27. Keras give input to intermediate layer and get final output
  28. Pyinstaller created exe file can not load a keras nn model
  29. Multi-input Multi-output Model with Keras Functional API
  30. Loading model with custom loss + keras

Leave a Comment