Specify connections in NN (in keras)

by

The simplest way I can think of, if you have this matrix correctly shaped, is to derive the Dense layer and simply add the matrix in the code multiplying the original weights:

class CustomConnected(Dense):

    def __init__(self,units,connections,**kwargs):

        #this is matrix A
        self.connections = connections                        

        #initalize the original Dense with all the usual arguments   
        super(CustomConnected,self).__init__(units,**kwargs)  


    def call(self,inputs):

        #change the kernel before calling the original call:
        self.kernel = self.kernel * self.connections

        #call the original calculations:
        super(CustomConnected,self).call(inputs)

Using:

model.add(CustomConnected(units,matrixA))
model.add(CustomConnected(hidden_dim2, matrixB,activation='tanh')) #can use all the other named parameters...

Notice that all the neurons/units have yet a bias added at the end. The argument use_bias=False will still work if you don’t want biases. You can also do exactly the same thing using a vector B, for instance, and mask the original biases with self.biases = self.biases * vectorB

Hint for testing: use different input and output dimensions, so you can be sure that your matrix A has the correct shape.

I just realized that my code is potentially buggy, because I’m changing a property that is used by the original Dense layer. If weird behaviors or messages appear, you can try another call method:

def call(self, inputs):
    output = K.dot(inputs, self.kernel * self.connections)
    if self.use_bias:
        output = K.bias_add(output, self.bias)
    if self.activation is not None:
        output = self.activation(output)
    return output

Where K comes from import keras.backend as K.

You may also go further and set a custom get_weights() method if you want to see the weights masked with your matrix. (This would not be necessary in the first approach above)

More Related Contents:

  1. TensorFlow and person recognition in video stream
  2. Keras Dense layer’s input is not flattened
  3. Keras : How should I prepare input data for RNN?
  4. How to Properly Combine TensorFlow’s Dataset API and Keras?
  5. What is the meaning of the “None” in model.summary of KERAS?
  6. Multivariate LSTM with missing values
  7. What are the differences between all these cross-entropy losses in Keras and TensorFlow?
  8. Negative dimension size caused by subtracting 3 from 1 for ‘conv2d_2/convolution’
  9. Does model.compile() initialize all the weights and biases in Keras (tensorflow backend)?
  10. Keras Loss Function with Additional Dynamic Parameter
  11. Using Gensim Fasttext model with LSTM nn in keras
  12. Create keras callback to save model predictions and targets for each batch during training
  13. Obtaining output of an Intermediate layer in TensorFlow/Keras
  14. tf.data with multiple inputs / outputs in Keras
  15. ValueError: Shapes (None, 1) and (None, 2) are incompatible
  16. How do I get the weights of a layer in Keras?
  17. Keras split train test set when using ImageDataGenerator
  18. In Keras, what exactly am I configuring when I create a stateful `LSTM` layer with N `units`?
  19. How to fix “AttributeError: module ‘tensorflow’ has no attribute ‘get_default_graph'”?
  20. Confusion about keras Model: __call__ vs. call vs. predict methods
  21. How can I convert a trained Tensorflow model to Keras?
  22. How to display custom images in TensorBoard using Keras?
  23. What is the difference between keras and tf.keras?
  24. Checkpointing keras model: TypeError: can’t pickle _thread.lock objects
  25. How to replace (or insert) intermediate layer in Keras model?
  26. Choosing number of Steps per Epoch
  27. How to compile Tensorflow with SSE4.2 and AVX instructions?
  28. Keras with TensorFlow backend not using GPU
  29. Could not install packages due to an EnvironmentError: [WinError 5] Access is denied:
  30. CUDA_ERROR_OUT_OF_MEMORY in tensorflow

Leave a Comment