Is there a way of determining how much GPU memory is in use by TensorFlow?

by

(1) There is some limited support with Timeline for logging memory allocations. Here is an example for its usage:

    run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
    run_metadata = tf.RunMetadata()
    summary, _ = sess.run([merged, train_step],
                          feed_dict=feed_dict(True),
                          options=run_options,
                          run_metadata=run_metadata)
    train_writer.add_run_metadata(run_metadata, 'step%03d' % i)
    train_writer.add_summary(summary, i)
    print('Adding run metadata for', i)
    tl = timeline.Timeline(run_metadata.step_stats)
    print(tl.generate_chrome_trace_format(show_memory=True))
    trace_file = tf.gfile.Open(name="timeline", mode="w")
    trace_file.write(tl.generate_chrome_trace_format(show_memory=True))

You can give this code a try with the MNIST example (mnist with summaries)

This will generate a tracing file named timeline, which you can open with chrome://tracing. Note that this only gives an approximated GPU memory usage statistics. It basically simulated a GPU execution, but doesn’t have access to the full graph metadata. It also can’t know how many variables have been assigned to the GPU.

(2) For a very coarse measure of GPU memory usage, nvidia-smi will show the total device memory usage at the time you run the command.

nvprof can show the on-chip shared memory usage and register usage at the CUDA kernel level, but doesn’t show the global/device memory usage.

Here is an example command: nvprof –print-gpu-trace matrixMul

And more details here:
http://docs.nvidia.com/cuda/profiler-users-guide/#abstract

More Related Contents:

  1. How to tell if tensorflow is using gpu acceleration from inside python shell?
  2. How to get current available GPUs in tensorflow?
  3. How do I use TensorFlow GPU?
  4. Clearing Tensorflow GPU memory after model execution
  5. Why is Tensorflow not recognizing my GPU after conda install?
  6. How to make Jupyter Notebook to run on GPU?
  7. AMD equivalent to NvOptimusEnablement
  8. Training a simple model in Tensorflow GPU slower than CPU
  9. How to print the value of a Tensor object in TensorFlow?
  10. How to choose cross-entropy loss in TensorFlow?
  11. Meaning of inter_op_parallelism_threads and intra_op_parallelism_threads
  12. Error running basic tensorflow example
  13. What is the meaning of the word logits in TensorFlow? [duplicate]
  14. What’s the difference between sparse_softmax_cross_entropy_with_logits and softmax_cross_entropy_with_logits?
  15. ValueError at /image/ Tensor Tensor(“activation_5/Softmax:0”, shape=(?, 4), dtype=float32) is not an element of this graph
  16. Tensorflow: ImportError: libcusolver.so.8.0: cannot open shared object file: No such file or directory
  17. expected ndim=3, found ndim=2
  18. NotImplementedError: Cannot convert a symbolic Tensor (lstm_2/strided_slice:0) to a numpy array. T
  19. Unbalanced data and weighted cross entropy
  20. Making predictions with a TensorFlow model
  21. How to count total number of trainable parameters in a tensorflow model?
  22. Tensor is not an element of this graph
  23. Keras LSTM input dimension setting
  24. Keras split train test set when using ImageDataGenerator
  25. Implementing contrastive loss and triplet loss in Tensorflow
  26. ‘Tensor’ object has no attribute ‘lower’
  27. What does batch, repeat, and shuffle do with TensorFlow Dataset?
  28. Google Colaboratory: misleading information about its GPU (only 5% RAM available to some users)
  29. How to replace (or insert) intermediate layer in Keras model?
  30. Tensorflow image reading & display

Leave a Comment