Keras: How to save model and continue training?

by

As it’s quite difficult to clarify where the problem is, I created a toy example from your code, and it seems to work alright.

import numpy as np
from numpy.testing import assert_allclose
from keras.models import Sequential, load_model
from keras.layers import LSTM, Dropout, Dense
from keras.callbacks import ModelCheckpoint

vec_size = 100
n_units = 10

x_train = np.random.rand(500, 10, vec_size)
y_train = np.random.rand(500, vec_size)

model = Sequential()
model.add(LSTM(n_units, input_shape=(None, vec_size), return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(n_units, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(n_units))
model.add(Dropout(0.2))
model.add(Dense(vec_size, activation='linear'))
model.compile(loss="mean_squared_error", optimizer="adam")

# define the checkpoint
filepath = "model.h5"
checkpoint = ModelCheckpoint(filepath, monitor="loss", verbose=1, save_best_only=True, mode="min")
callbacks_list = [checkpoint]

# fit the model
model.fit(x_train, y_train, epochs=5, batch_size=50, callbacks=callbacks_list)

# load the model
new_model = load_model(filepath)
assert_allclose(model.predict(x_train),
                new_model.predict(x_train),
                1e-5)

# fit the model
checkpoint = ModelCheckpoint(filepath, monitor="loss", verbose=1, save_best_only=True, mode="min")
callbacks_list = [checkpoint]
new_model.fit(x_train, y_train, epochs=5, batch_size=50, callbacks=callbacks_list)

The loss continues to decrease after model loading. (restarting python also gives no problem)

Using TensorFlow backend.
Epoch 1/5
500/500 [==============================] - 2s - loss: 0.3216     Epoch 00000: loss improved from inf to 0.32163, saving model to model.h5
Epoch 2/5
500/500 [==============================] - 0s - loss: 0.2923     Epoch 00001: loss improved from 0.32163 to 0.29234, saving model to model.h5
Epoch 3/5
500/500 [==============================] - 0s - loss: 0.2542     Epoch 00002: loss improved from 0.29234 to 0.25415, saving model to model.h5
Epoch 4/5
500/500 [==============================] - 0s - loss: 0.2086     Epoch 00003: loss improved from 0.25415 to 0.20860, saving model to model.h5
Epoch 5/5
500/500 [==============================] - 0s - loss: 0.1725     Epoch 00004: loss improved from 0.20860 to 0.17249, saving model to model.h5

Epoch 1/5
500/500 [==============================] - 0s - loss: 0.1454     Epoch 00000: loss improved from inf to 0.14543, saving model to model.h5
Epoch 2/5
500/500 [==============================] - 0s - loss: 0.1289     Epoch 00001: loss improved from 0.14543 to 0.12892, saving model to model.h5
Epoch 3/5
500/500 [==============================] - 0s - loss: 0.1169     Epoch 00002: loss improved from 0.12892 to 0.11694, saving model to model.h5
Epoch 4/5
500/500 [==============================] - 0s - loss: 0.1097     Epoch 00003: loss improved from 0.11694 to 0.10971, saving model to model.h5
Epoch 5/5
500/500 [==============================] - 0s - loss: 0.1057     Epoch 00004: loss improved from 0.10971 to 0.10570, saving model to model.h5

BTW, redefining the model followed by load_weight() definitely won’t work, because save_weight() and load_weight() does not save/load the optimizer.

More Related Contents:

  1. Keras Dense layer’s input is not flattened
  2. How to get reproducible results in keras
  3. Tensorflow – ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type float)
  4. Keras + Tensorflow and Multiprocessing in Python
  5. Getting gradient of model output w.r.t weights using Keras
  6. Proper way to feed time-series data to stateful LSTM?
  7. How to use a Keras RNN model to forecast for future dates or events?
  8. TypeError: __array__() takes 1 positional argument but 2 were given (Image classification Keras)
  9. Tensorflow 2.0 – AttributeError: module ‘tensorflow’ has no attribute ‘Session’
  10. Keras – Plot training, validation and test set accuracy
  11. Neural Network LSTM input shape from dataframe
  12. Passing output of a CNN to BILSTM
  13. Get class labels from Keras functional model
  14. How do I check if keras is using gpu version of tensorflow?
  15. How does Keras handle multilabel classification?
  16. Keras: .predict returns percentages instead of classes
  17. Keras Custom loss function to pass arguments other than y_true and y_pred
  18. Save and load weights in keras
  19. How to fix ‘Object arrays cannot be loaded when allow_pickle=False’ for imdb.load_data() function?
  20. Unknown initializer: GlorotUniform when loading Keras model
  21. Tensorflow : logits and labels must have the same first dimension
  22. How to add and remove new layers in keras after loading weights?
  23. How to log Keras loss output to a file
  24. How to fix “AttributeError: module ‘tensorflow’ has no attribute ‘get_default_graph'”?
  25. ‘Tensor’ object has no attribute ‘lower’
  26. ValueError: Output tensors to a Model must be the output of a TensorFlow `Layer`
  27. Strange behaviour of the loss function in keras model, with pretrained convolutional base
  28. Keras give input to intermediate layer and get final output
  29. Loading model with custom loss + keras
  30. Invalid Argument Error / Graph Execution Error

Leave a Comment