Tensorflow and Multiprocessing: Passing Sessions

by

You can’t use Python multiprocessing to pass a TensorFlow Session into a multiprocessing.Pool in the straightfoward way because the Session object can’t be pickled (it’s fundamentally not serializable because it may manage GPU memory and state like that).

I’d suggest parallelizing the code using actors, which are essentially the parallel computing analog of “objects” and use used to manage state in the distributed setting.

Ray is a good framework for doing this. You can define a Python class which manages the TensorFlow Session and exposes a method for running your simulation.

import ray
import tensorflow as tf

ray.init()

@ray.remote
class Simulator(object):
    def __init__(self):
        self.sess = tf.Session()
        self.simple_model = tf.constant([1.0])

    def simulate(self):
        return self.sess.run(self.simple_model)

# Create two actors.
simulators = [Simulator.remote() for _ in range(2)]

# Run two simulations in parallel.
results = ray.get([s.simulate.remote() for s in simulators])

Here are a few more examples of parallelizing TensorFlow with Ray.

See the Ray documentation. Note that I’m one of the Ray developers.

More Related Contents:

  1. Shared-memory objects in multiprocessing
  2. What are the differences between the threading and multiprocessing modules?
  3. Sharing a result queue among several processes
  4. multiprocessing: Understanding logic behind `chunksize`
  5. multiprocessing.dummy in Python is not utilising 100% cpu
  6. Meaning of inter_op_parallelism_threads and intra_op_parallelism_threads
  7. Same output in different workers in multiprocessing
  8. Using the multiprocessing module for cluster computing
  9. Poor scaling of multiprocessing Pool.map() on a list of large objects: How to achieve better parallel scaling in python?
  10. Python multiprocessing.Pool: AttributeError
  11. Modify object in python multiprocessing
  12. multiprocessing pool example does not work and freeze the kernel
  13. Keras not training on entire dataset
  14. FailedPreconditionError: Attempting to use uninitialized in Tensorflow
  15. Concurrent.futures vs Multiprocessing in Python 3
  16. Tensorflow error : DLL load failed: The specified procedure could not be found
  17. can’t import tensorflow in python, windows 10 64 bit
  18. How to set adaptive learning rate for GradientDescentOptimizer?
  19. multiprocessing.pool.MaybeEncodingError: ‘TypeError(“cannot serialize ‘_io.BufferedReader’ object”,)’
  20. High Memory Usage Using Python Multiprocessing
  21. TensorFlow ValueError: Cannot feed value of shape (64, 64, 3) for Tensor u’Placeholder:0′, which has shape ‘(?, 64, 64, 3)’
  22. TensorFlow – numpy-like tensor indexing
  23. Can multiprocessing Process class be run from IDLE
  24. Using 100% of all cores with the multiprocessing module
  25. Python multiprocessing memory usage
  26. How to fix “ResourceExhaustedError: OOM when allocating tensor”
  27. How to limit number of CPU’s used by a python script w/o terminal or multiprocessing library?
  28. Multiprocessing pool with an iterator
  29. How is the Keras Conv1D input specified? I seem to be lacking a dimension
  30. multiprocessing: How can I ʀᴇʟɪᴀʙʟʏ redirect stdout from a child process?

Leave a Comment