Tensorflow crashes with CUBLAS_STATUS_ALLOC_FAILED

by

For TensorFlow 2.2 none of the other answers worked when the CUBLAS_STATUS_ALLOC_FAILED problem was encountered. Found a solution on https://www.tensorflow.org/guide/gpu:

import tensorflow as tf
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
    try:
        # Currently, memory growth needs to be the same across GPUs
        for gpu in gpus:
            tf.config.experimental.set_memory_growth(gpu, True)
        logical_gpus = tf.config.experimental.list_logical_devices('GPU')
        print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
    except RuntimeError as e:
        # Memory growth must be set before GPUs have been initialized
        print(e)

I ran this code before any further calculations are made and found that the same code that produced CUBLAS error before now worked in same session. The sample code above is a specific example that sets the memory growth across a number of physical GPUs but it also solves the memory expansion problem.

More Related Contents:

  1. TensorFlow and person recognition in video stream
  2. How to compile Tensorflow with SSE4.2 and AVX instructions?
  3. How to understand static shape and dynamic shape in TensorFlow?
  4. Meaning of buffer_size in Dataset.map , Dataset.prefetch and Dataset.shuffle
  5. Can I use TensorBoard with Google Colab?
  6. How do I disable TensorFlow’s eager execution?
  7. After building TensorFlow from source, seeing libcudart.so and libcudnn errors
  8. Convert a graph proto (pb/pbtxt) to a SavedModel for use in TensorFlow Serving or Cloud ML Engine
  9. How to create dataset in the same format as the FSNS dataset?
  10. Specify connections in NN (in keras)
  11. What are the differences between all these cross-entropy losses in Keras and TensorFlow?
  12. Split a dataset created by Tensorflow dataset API in to Train and Test?
  13. Does model.compile() initialize all the weights and biases in Keras (tensorflow backend)?
  14. Keras Loss Function with Additional Dynamic Parameter
  15. TensorFlow: How to measure how much GPU memory each tensor takes?
  16. How to convert .pb to TFLite format?
  17. How to select rows from a 3-D Tensor in TensorFlow?
  18. Working with multiple graphs in TensorFlow
  19. Simple way to visualize a TensorFlow graph in Jupyter?
  20. ValueError: Shapes (None, 1) and (None, 2) are incompatible
  21. Update only part of the word embedding matrix in Tensorflow
  22. Could not install packages due to an EnvironmentError: [WinError 5] Access is denied:
  23. CUDA_ERROR_OUT_OF_MEMORY in tensorflow
  24. Convert between NHWC and NCHW in TensorFlow
  25. How to properly use tf.metrics.accuracy?
  26. How can I convert a trained Tensorflow model to Keras?
  27. ImportError: libcudnn when running a TensorFlow program
  28. Google Storage (gs) wrapper file input/out for Cloud ML?
  29. parallelising tf.data.Dataset.from_generator
  30. Choosing number of Steps per Epoch

Leave a Comment