Is it possible to multiprocess a function that returns something in Python?

by

You are looking to do some embarrassingly parallel work using multiple processes, so why not use a Pool? A Pool will take care of starting up the processes, retrieving the results, and returning the results to you.

I use pathos, which has a fork of multiprocessing, because it has much better serialization than the version that standard library provides.

(.py) file

from pathos.multiprocessing import ProcessingPool as Pool

def foo(obj1, obj2):
    a = obj1.x**2
    b = obj2.x**2
    return a,b

class Bar(object):
    def __init__(self, x):
        self.x = x

Pool().map(foo, [Bar(1),Bar(2),Bar(3)], [Bar(4),Bar(5),Bar(6)])

Result

[(1, 16), (4, 25), (9, 36)]

And you see that foo takes two arguments, and returns a tuple of two objects. The map method of Pool submits foo to the underlying processes and returns the result as res.

You can get pathos here: https://github.com/uqfoundation

More Related Contents:

  1. How to run functions in parallel?
  2. multiprocessing.Pool: When to use apply, apply_async or map?
  3. Shared-memory objects in multiprocessing
  4. Shared memory in multiprocessing
  5. Dead simple example of using Multiprocessing Queue, Pool and Locking
  6. Shared variable in python’s multiprocessing
  7. multiprocessing.Pool: What’s the difference between map_async and imap?
  8. How to use a multiprocessing.Manager()?
  9. Solving embarassingly parallel problems using Python multiprocessing
  10. Log output of multiprocessing.Process
  11. multiprocessing vs multithreading vs asyncio in Python 3
  12. pandas multiprocessing apply
  13. Why does Python’s multiprocessing module import __main__ when starting a new process on Windows?
  14. Multiprocessing : use tqdm to display a progress bar
  15. Appending to the same list from different processes using multiprocessing
  16. Using the multiprocessing module for cluster computing
  17. multiprocessing.Pool() slower than just using ordinary functions
  18. Why can I pass an instance method to multiprocessing.Process, but not a multiprocessing.Pool?
  19. Chrome crashes after several hours while multiprocessing using Selenium through Python
  20. How to let Pool.map take a lambda function
  21. Python Multiprocessing error: AttributeError: module ‘__main__’ has no attribute ‘__spec__’
  22. Python multiprocessing application is getting stuck in docker container
  23. Can I get a return value from multiprocessing.Process?
  24. How to utilize all cores with python multiprocessing
  25. Tensorflow and Multiprocessing: Passing Sessions
  26. How to synchronize a python dict with multiprocessing
  27. How can I share a class between processes?
  28. Multiprocessing useless with urllib2?
  29. Python multiprocessing linux windows difference
  30. python multiprocessing in Jupyter on Windows: AttributeError: Can’t get attribute “abc”

Leave a Comment