Python multiprocessing – tracking the process of pool.map operation

by

Note that I’m using pathos.multiprocessing instead of multiprocessing. It’s just a fork of multiprocessing that enables you to do map functions with multiple inputs, has much better serialization, and allows you to execute map calls anywhere (not just in __main__). You could use multiprocessing to do the below as well, however the code would be very slightly different.

If you use an iterated map function, it’s pretty easy to keep track of progress.

from pathos.multiprocessing import ProcessingPool as Pool
def simFunction(x,y):
  import time
  time.sleep(2)
  return x**2 + y
 
x,y = range(100),range(-100,100,2)
res = Pool().imap(simFunction, x,y)
with open('results.txt', 'w') as out:
  for i in x:
    out.write("%s\n" % res.next())
    if i%10 is 0:
      print "%s of %s simulated" % (i, len(x))

0 of 100 simulated
10 of 100 simulated
20 of 100 simulated
30 of 100 simulated
40 of 100 simulated
50 of 100 simulated
60 of 100 simulated
70 of 100 simulated
80 of 100 simulated
90 of 100 simulated

Or, you can use an asynchronous map. Here I’ll do things a little differently, just to mix it up.

import time
res = Pool().amap(simFunction, x,y)
while not res.ready():
  print "waiting..."
  time.sleep(5)
 

waiting...
waiting...
waiting...
waiting...

res.get()
[-100, -97, -92, -85, -76, -65, -52, -37, -20, -1, 20, 43, 68, 95, 124, 155, 188, 223, 260, 299, 340, 383, 428, 475, 524, 575, 628, 683, 740, 799, 860, 923, 988, 1055, 1124, 1195, 1268, 1343, 1420, 1499, 1580, 1663, 1748, 1835, 1924, 2015, 2108, 2203, 2300, 2399, 2500, 2603, 2708, 2815, 2924, 3035, 3148, 3263, 3380, 3499, 3620, 3743, 3868, 3995, 4124, 4255, 4388, 4523, 4660, 4799, 4940, 5083, 5228, 5375, 5524, 5675, 5828, 5983, 6140, 6299, 6460, 6623, 6788, 6955, 7124, 7295, 7468, 7643, 7820, 7999, 8180, 8363, 8548, 8735, 8924, 9115, 9308, 9503, 9700, 9899]

Either an iterated or asynchronous map will enable you to write whatever code you want to do better process tracking. For example, pass a unique “id” to each job, and watch which come back, or have each job return it’s process id. There are lots of ways to track progress and processes… but the above should give you a start.

You can get pathos here.

More Related Contents:

  1. Can’t pickle when using multiprocessing Pool.map()
  2. Keyboard Interrupts with python’s multiprocessing Pool
  3. Python Process Pool non-daemonic?
  4. Combine Pool.map with shared memory Array in Python multiprocessing
  5. Filling a queue and managing multiprocessing in python
  6. Multiprocessing.Pool makes Numpy matrix multiplication slower
  7. Can I use a multiprocessing Queue in a function called by Pool.imap?
  8. multiprocessing.Pool() slower than just using ordinary functions
  9. How do you pass a Queue reference to a function managed by pool.map_async()?
  10. Passing multiple parameters to pool.map() function in Python [duplicate]
  11. Python Multiprocess Pool. How to exit the script when one of the worker process determines no more work needs to be done?
  12. How to let Pool.map take a lambda function
  13. Python: Writing to a single file with queue while using multiprocessing Pool
  14. How to use Python multiprocessing Pool.map to fill numpy array in a for loop
  15. How to run functions in parallel?
  16. Dead simple example of using Multiprocessing Queue, Pool and Locking
  17. Share Large, Read-Only Numpy Array Between Multiprocessing Processes
  18. multiprocessing.Pool: What’s the difference between map_async and imap?
  19. Log output of multiprocessing.Process
  20. pandas multiprocessing apply
  21. Why does Python’s multiprocessing module import __main__ when starting a new process on Windows?
  22. Using the multiprocessing module for cluster computing
  23. Python Multiprocessing error: AttributeError: module ‘__main__’ has no attribute ‘__spec__’
  24. Can I get a return value from multiprocessing.Process?
  25. How to utilize all cores with python multiprocessing
  26. Tensorflow and Multiprocessing: Passing Sessions
  27. How can I share a class between processes?
  28. Multiprocessing useless with urllib2?
  29. Python multiprocessing linux windows difference
  30. python multiprocessing in Jupyter on Windows: AttributeError: Can’t get attribute “abc”

Leave a Comment