You can’t use Python multiprocessing to pass a TensorFlow Session into a multiprocessing.Pool in the straightfoward way because the Session object can’t be pickled (it’s fundamentally not serializable because it may manage GPU memory and state like that). I’d suggest parallelizing the code using actors, which are essentially the parallel computing analog of “objects” and … Read more
Too much piling up here to address in comments, so, where mp is multiprocessing: mp.cpu_count() should return the number of processors. But test it. Some platforms are funky, and this info isn’t always easy to get. Python does the best it can. If you start 24 processes, they’ll do exactly what you tell them to … Read more
If you want a very easy solution, there isn’t one. However, there is a solution that has the multiprocessing interface — pathos — which has the ability to establish connections to remote servers through a parallel map, and to do multiprocessing. If you want to have a ssh-tunneled connection, you can do that… or if … Read more
The MonteCarlo objects have been pickled and sent to child processes to be run – the .results attribute in this process isn’t populated because the local mc has never been run. If you create a multiprocessing.Queue, you can pass that into each MonteCarlo job, and when it finishes it should put the result in there. … Read more
I think that the answer is “Yes”. This probably as good as you can do in Java using system independent code. But be aware that even this is only relatively system independent. For example, in some systems: the JAVA_HOME variable may not have been set, the command name used to launch a JVM might be … Read more
multiprocessing.Lock is implemented using a Semaphore object provided by the OS. On Linux, the child just inherits a handle to the Semaphore from the parent via os.fork. This isn’t a copy of the semaphore; it’s actually inheriting the same handle the parent has, the same way file descriptors can be inherited. Windows on the other … Read more
multiprocessing.Lock is implemented using a Semaphore object provided by the OS. On Linux, the child just inherits a handle to the Semaphore from the parent via os.fork. This isn’t a copy of the semaphore; it’s actually inheriting the same handle the parent has, the same way file descriptors can be inherited. Windows on the other … Read more
It might be most sensible to use multiprocessing.Pool which produces a pool of worker processes based on the max number of cores available on your system, and then basically feeds tasks in as the cores become available. The example from the standard docs (http://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers) shows that you can also manually set the number of cores: … Read more
Two approaches: try concurrent.futures process executor Different libraries handle cases like these differently. Might fail more gracefully under your specific circumstances. psutils and manual memory allocation logging If you are building this into a production process, this is a must given errors you’ve encountered so far.
I just had the similar problem on the project with 3 servers/dispatchers running each on its own CPU core, spawning highly CPU intensive workers – which, at the same time, were designed to use specific core(s) they were running on to its fullest potential. I needed to assure that such worker process would never be … Read more