Keras + Tensorflow and Multiprocessing in Python

by

From my experience – the problem lies in loading Keras to one process and then spawning a new process when the keras has been loaded to your main environment. But for some applications (like e.g. training a mixture of Kerasmodels) it’s simply better to have all of this things in one process. So what I advise is the following (a little bit cumbersome – but working for me) approach:

  1. DO NOT LOAD KERAS TO YOUR MAIN ENVIRONMENT. If you want to load Keras / Theano / TensorFlow do it only in the function environment. E.g. don’t do this:

    import keras

def training_function(...):
    ...

    but do the following:

    def training_function(...):
    import keras
    ...

  2. Run work connected with each model in a separate process: I’m usually creating workers which are making the job (like e.g. training, tuning, scoring) and I’m running them in separate processes. What is nice about it that whole memory used by this process is completely freed when your process is done. This helps you with loads of memory problems which you usually come across when you are using multiprocessing or even running multiple models in one process. So this looks e.g. like this:

    def _training_worker(train_params):
    import keras
    model = obtain_model(train_params)
    model.fit(train_params)
    send_message_to_main_process(...)

def train_new_model(train_params):
    training_process = multiprocessing.Process(target=_training_worker, args = train_params)
    training_process.start()
    get_message_from_training_process(...)
    training_process.join()

Different approach is simply preparing different scripts for different model actions. But this may cause memory errors especially when your models are memory consuming. NOTE that due to this reason it’s better to make your execution strictly sequential.

More Related Contents:

  1. Negative dimension size caused by subtracting 3 from 1 for ‘conv2d_2/convolution’
  2. Loading a trained Keras model and continue training
  3. How do I check if keras is using gpu version of tensorflow?
  4. Multiple outputs in Keras
  5. TensorFlow Only running on 1/32 of the Training data provided [duplicate]
  6. Make a custom loss function in keras
  7. How to export Keras .h5 to tensorflow .pb?
  8. How to choose cross-entropy loss in TensorFlow?
  9. Could not load dynamic library ‘cudart64_101.dll’ on tensorflow CPU-only installation
  10. How does Keras calculate the accuracy?
  11. NotImplementedError: Layers with arguments in `__init__` must override `get_config`
  12. NaN loss when training regression network
  13. Keras replacing input layer
  14. Tensorflow: How to write op with gradient in python?
  15. Keras Sequential model with multiple inputs
  16. expected ndim=3, found ndim=2
  17. This model has not yet been built error on model.summary()
  18. How to get other metrics in Tensorflow 2.0 (not only accuracy)?
  19. How can I use a pre-trained neural network with grayscale images?
  20. Removing then Inserting a New Middle Layer in a Keras Model
  21. Custom loss function in Keras
  22. Save and load model optimizer state
  23. Keras misinterprets training data shape
  24. Where do I call the BatchNormalization function in Keras?
  25. How do I get the weights of a layer in Keras?
  26. Tensor is not an element of this graph
  27. ValueError: Input 0 of layer sequential is incompatible with the layer: : expected min_ndim=4, found ndim=3. Full shape received: [8, 28, 28]
  28. Keras LSTM input dimension setting
  29. Tensorflow One Hot Encoder?
  30. Unexpected keyword argument ‘ragged’ in Keras

Leave a Comment