Keras Dense layer’s input is not flattened

by

Currently, contrary to what has been stated in documentation, the Dense layer is applied on the last axis of input tensor:

Contrary to the documentation, we don’t actually flatten it. It’s
applied on the last axis independently.

In other words, if a Dense layer with m units is applied on an input tensor of shape (n_dim1, n_dim2, ..., n_dimk) it would have an output shape of (n_dim1, n_dim2, ..., m).

As a side note: this makes TimeDistributed(Dense(...)) and Dense(...) equivalent to each other.

Another side note: be aware that this has the effect of shared weights. For example, consider this toy network:

model = Sequential()
model.add(Dense(10, input_shape=(20, 5)))

model.summary()

The model summary:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 20, 10)            60        
=================================================================
Total params: 60
Trainable params: 60
Non-trainable params: 0
_________________________________________________________________

As you can see the Dense layer has only 60 parameters. How? Each unit in the Dense layer is connected to the 5 elements of each row in the input with the same weights, therefore 10 * 5 + 10 (bias params per unit) = 60.

Update. Here is a visual illustration of the example above:

Visual illustration of applying a Dense layer on an input with two or more dimensions in Keras

More Related Contents:

  1. Make a custom loss function in keras
  2. Keras not training on entire dataset
  3. Deep-Learning Nan loss reasons
  4. How to apply gradient clipping in TensorFlow?
  5. Can Keras with Tensorflow backend be forced to use CPU or GPU at will?
  6. Proper way to feed time-series data to stateful LSTM?
  7. ValueError at /image/ Tensor Tensor(“activation_5/Softmax:0”, shape=(?, 4), dtype=float32) is not an element of this graph
  8. Keras custom loss function: Accessing current input pattern
  9. Negative dimension size caused by subtracting 3 from 1 for ‘conv2d_2/convolution’
  10. How to get other metrics in Tensorflow 2.0 (not only accuracy)?
  11. How can I use a pre-trained neural network with grayscale images?
  12. Removing then Inserting a New Middle Layer in a Keras Model
  13. How to know if underfitting or overfitting is occuring?
  14. Save and load model optimizer state
  15. Unknown initializer: GlorotUniform when loading Keras model
  16. How do I get the weights of a layer in Keras?
  17. How to create a keras layer with a custom gradient in TF2.0?
  18. How to add and remove new layers in keras after loading weights?
  19. How to fix “AttributeError: module ‘tensorflow’ has no attribute ‘get_default_graph'”?
  20. ‘Tensor’ object has no attribute ‘lower’
  21. ValueError: Output tensors to a Model must be the output of a TensorFlow `Layer`
  22. How to fix “ResourceExhaustedError: OOM when allocating tensor”
  23. Keras give input to intermediate layer and get final output
  24. How is the Keras Conv1D input specified? I seem to be lacking a dimension
  25. Does Any one got “AttributeError: ‘str’ object has no attribute ‘decode’ ” , while Loading a Keras Saved Model
  26. ValueError: Error when checking target: expected model_2 to have shape (None, 252, 252, 1) but got array with shape (300, 128, 128, 3)
  27. AttributeError: module ‘tensorflow.python.keras.utils.generic_utils’ has no attribute ‘populate_dict_with_module_objects’
  28. What is the role of TimeDistributed layer in Keras?
  29. keras: how to save the training history attribute of the history object
  30. Unexpected keyword argument ‘ragged’ in Keras

Leave a Comment