Tensorflow : Memory leak even while closing Session?

by

TL;DR: Closing a session does not free the tf.Graph data structure in your Python program, and if each iteration of the loop adds nodes to the graph, you’ll have a leak.

Since your function feedForwardStep creates new TensorFlow operations, and you call it within the for loop, then there is a leak in your code—albeit a subtle one.

Unless you specify otherwise (using a with tf.Graph().as_default(): block), all TensorFlow operations are added to a global default graph. This means that every call to tf.constant(), tf.matmul(), tf.Variable() etc. adds objects to a global data structure. There are two ways to avoid this:

  1. Structure your program so that you build the graph once, then use tf.placeholder() ops to feed in different values in each iteration. You mention in your question that this might not be possible.

  2. Explicitly create a new graph in each for loop. This might be necessary if the structure of the graph depends on the data available in the current iteration. You would do this as follows:

    for step in xrange(200):
    with tf.Graph().as_default(), tf.Session() as sess:
        # Remainder of loop body goes here.

    Note that in this version, you cannot use Tensor or Operation objects from a previous iteration. (For example, it’s not clear from your code snippet where test comes from.)

More Related Contents:

  1. Tensor is not an element of this graph
  2. What and where are the stack and heap?
  3. NSMutableArray can not release objects after invoke removeAllObjects
  4. What is private bytes, virtual bytes, working set?
  5. Python memory leaks [closed]
  6. Does Rust free up the memory of overwritten variables?
  7. What strategies and tools are useful for finding memory leaks in .NET?
  8. Large Object Heap Fragmentation
  9. Memory leak detection tools
  10. What are the differences between virtual memory and physical memory?
  11. iPhone OS Memory Warnings. What Do The Different Levels Mean?
  12. memory limit in Node.js (and chrome V8)
  13. .NET Memory issues loading ~40 images, memory not reclaimed, potentially due to LOH fragmentation
  14. Circular References in Java
  15. MATLAB’s Garbage Collector?
  16. Does calling free or delete ever release memory back to the “system”
  17. gfortran for dummies: What does mcmodel=medium do exactly?
  18. What does tf.nn.conv2d do in tensorflow?
  19. Perl memory usage profiling and leak detection?
  20. Tensorflow: How to write op with gradient in python?
  21. Multivariate LSTM with missing values
  22. Negative dimension size caused by subtracting 3 from 1 for ‘conv2d_2/convolution’
  23. What do I need to do before deleting elements in a vector of pointers to dynamically allocated objects?
  24. Is freeing handled differently for small/large numpy arrays?
  25. How to count total number of trainable parameters in a tensorflow model?
  26. How to get memory size of variable?
  27. What is right batch normalization function in Tensorflow?
  28. iPhone development: pointer being freed was not allocated
  29. Do unclosed streams cause memory leaks in java?
  30. C++ is it necessary to delete dynamically allocated objects at the end of the main scope?

Leave a Comment