How to interpret MSE in Keras Regressor

by

I apologise for sounding silly as I am starting out!

Do not; this is a subtle issue of great importance, which is usually (and regrettably) omitted in tutorials and introductory expositions.

Unfortunately, it is not as simple as taking the square root of the inverse-transformed MSE, but it is not that complicated either; essentially what you have to do is:

  1. Transform back your predictions to the initial scale of the original data
  2. Get the MSE between these invert-transformed predictions and the original data
  3. Take the square root of the result

in order to get a performance indicator of your model that will be meaningful in the business context of your problem (e.g. US dollars here).

Let’s see a quick example with toy data, omitting the model itself (which is irrelevant here, and in fact can be any regression model – not only a Keras one):

from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error
import numpy as np

# toy data
X = np.array([[1,2], [3,4], [5,6], [7,8], [9,10]])
Y = np.array([3, 4, 5, 6, 7])

# feature scaling
sc_X = StandardScaler()
X_train = sc_X.fit_transform(X)

# outcome scaling:
sc_Y = StandardScaler()
Y_train = sc_Y.fit_transform(Y.reshape(-1, 1))
Y_train
# array([[-1.41421356],
#        [-0.70710678],
#        [ 0.        ],
#        [ 0.70710678],
#        [ 1.41421356]])

Now, let’s say that we fit our Keras model (not shown here) using the scaled sets X_train and Y_train, and get predictions on the training set:

prediction = model.predict(X_train) # scaled inputs here
print(prediction)
# [-1.4687586  -0.6596055   0.14954728  0.95870024  1.001172  ]

The MSE reported by Keras is actually the scaled MSE, i.e.:

MSE_scaled = mean_squared_error(Y_train, prediction)
MSE_scaled
# 0.052299712818541934

while the 3 steps I have described above are simply:

MSE = mean_squared_error(Y, sc_Y.inverse_transform(prediction))  # first 2 steps, combined
MSE
# 0.10459946572909758
np.sqrt(MSE)  # 3rd step
# 0.323418406602187

So, in our case, if our initial Y were US dollars, the actual error in the same units (dollars) would be 0.32 (dollars).

Notice how the naive approach of inverse-transforming the scaled MSE would give a very different (and incorrect) result:

np.sqrt(sc_Y.inverse_transform([MSE_scaled]))
# array([2.25254588])

More Related Contents:

  1. What function defines accuracy in Keras when the loss is mean squared error (MSE)?
  2. Early stopping with Keras and sklearn GridSearchCV cross-validation
  3. Is this possible to predict the lottery numbers (not the most accurate)? [closed]
  4. Why binary_crossentropy and categorical_crossentropy give different performances for the same problem?
  5. Loss & accuracy – Are these reasonable learning curves?
  6. Why do we have to normalize the input for an artificial neural network? [closed]
  7. Many to one and many to many LSTM examples in Keras
  8. Getting a low ROC AUC score but a high accuracy
  9. Why does one hot encoding improve machine learning performance? [closed]
  10. How to calculate optimal batch size
  11. What is the difference between loss function and metric in Keras? [duplicate]
  12. Keras Text Preprocessing – Saving Tokenizer object to file for scoring
  13. ResNet: 100% accuracy during training, but 33% prediction accuracy with the same data
  14. Save MinMaxScaler model in sklearn
  15. Is deep learning bad at fitting simple non linear functions outside training scope (extrapolating)?
  16. confusion matrix error “Classification metrics can’t handle a mix of multilabel-indicator and multiclass targets”
  17. How to put more weight on certain features in machine learning?
  18. Can Keras deal with input images with different size?
  19. TimeDistributed(Dense) vs Dense in Keras – Same number of parameters
  20. Keras: model.evaluate vs model.predict accuracy difference in multi-class NLP task
  21. What is the difference between Keras model.evaluate() and model.predict()?
  22. Using pre-trained word2vec with LSTM for word generation
  23. Scikits-Learn RandomForrest trained on 64bit python wont open on 32bit python
  24. Can I send callbacks to a KerasClassifier?
  25. Scikit learn – fit_transform on the test set
  26. How can I get biases from a trained model in Keras?
  27. How should I teach machine learning algorithm using data with big disproportion of classes? (SVM)
  28. Under what parameters are SVC and LinearSVC in scikit-learn equivalent?
  29. Artificial Neural Network Toplogy
  30. Save and load model optimizer state

Leave a Comment