Why should weights of Neural Networks be initialized to random numbers? [closed]

by

Breaking symmetry is essential here, and not for the reason of performance. Imagine first 2 layers of multilayer perceptron (input and hidden layers):

enter image description here

During forward propagation each unit in hidden layer gets signal:

enter image description here

That is, each hidden unit gets sum of inputs multiplied by the corresponding weight.

Now imagine that you initialize all weights to the same value (e.g. zero or one). In this case, each hidden unit will get exactly the same signal. E.g. if all weights are initialized to 1, each unit gets signal equal to sum of inputs (and outputs sigmoid(sum(inputs))). If all weights are zeros, which is even worse, every hidden unit will get zero signal. No matter what was the input – if all weights are the same, all units in hidden layer will be the same too.

This is the main issue with symmetry and reason why you should initialize weights randomly (or, at least, with different values). Note, that this issue affects all architectures that use each-to-each connections.

More Related Contents:

  1. Is this possible to predict the lottery numbers (not the most accurate)? [closed]
  2. Artificial Neural Network Toplogy
  3. What is the role of the bias in neural networks? [closed]
  4. Clarification on a Neural Net that plays Snake
  5. Common causes of nans during training of neural networks
  6. How to interpret caffe log with debug_info?
  7. Epoch vs Iteration when training neural networks [closed]
  8. What is `weight_decay` meta parameter in Caffe?
  9. Cost function training target versus accuracy desired goal
  10. How to calculate optimal batch size
  11. How to interpret loss and accuracy for a machine learning model [closed]
  12. Caffe: What can I do if only a small batch fits into memory?
  13. What is `lr_policy` in Caffe?
  14. Machine Learning Two class classification [closed]
  15. How to interpret data output from Weka?
  16. Loss & accuracy – Are these reasonable learning curves?
  17. Why do we have to normalize the input for an artificial neural network? [closed]
  18. Many to one and many to many LSTM examples in Keras
  19. Caffe | solver.prototxt values setting strategy
  20. What is the difference between loss function and metric in Keras? [duplicate]
  21. Unable to approximate the sine function using a neural network
  22. How to create a new gym environment in OpenAI?
  23. Is deep learning bad at fitting simple non linear functions outside training scope (extrapolating)?
  24. multi-layer perceptron (MLP) architecture: criteria for choosing number of hidden layers and size of the hidden layer? [closed]
  25. TimeDistributed(Dense) vs Dense in Keras – Same number of parameters
  26. How to create caffe.deploy from train.prototxt
  27. Neural network XOR gate not learning
  28. What is the difference between supervised learning and unsupervised learning? [closed]
  29. What is the difference between Keras model.evaluate() and model.predict()?
  30. Using pre-trained word2vec with LSTM for word generation

Leave a Comment