Python sharing a lock between processes

by

You can’t pass normal multiprocessing.Lock objects to Pool methods, because they can’t be pickled. There are two ways to get around this. One is to create Manager() and pass a Manager.Lock():

def main():
    iterable = [1, 2, 3, 4, 5]
    pool = multiprocessing.Pool()
    m = multiprocessing.Manager()
    l = m.Lock()
    func = partial(target, l)
    pool.map(func, iterable)
    pool.close()
    pool.join()

This is a little bit heavyweight, though; using a Manager requires spawning another process to host the Manager server. And all calls to acquire/release the lock have to be sent to that server via IPC.

The other option is to pass the regular multiprocessing.Lock() at Pool creation time, using the initializer kwarg. This will make your lock instance global in all the child workers:

def target(iterable_item):
    for item in items:
        # Do cool stuff
        if (... some condition here ...):
            lock.acquire()
            # Write to stdout or logfile, etc.
            lock.release()
def init(l):
    global lock
    lock = l

def main():
    iterable = [1, 2, 3, 4, 5]
    l = multiprocessing.Lock()
    pool = multiprocessing.Pool(initializer=init, initargs=(l,))
    pool.map(target, iterable)
    pool.close()
    pool.join()

The second solution has the side-effect of no longer requiring partial.

More Related Contents:

  1. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  2. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  3. Python multiprocessing PicklingError: Can’t pickle
  4. Using global variables between files?
  5. Why does multiprocessing use only a single core after I import numpy?
  6. Sharing a result queue among several processes
  7. Keyboard Interrupts with python’s multiprocessing Pool
  8. multiprocessing: Understanding logic behind `chunksize`
  9. Dead simple example of using Multiprocessing Queue, Pool and Locking
  10. Multiprocessing: How to use Pool.map on a function defined in a class?
  11. multiprocessing in python – sharing large object (e.g. pandas dataframe) between multiple processes
  12. pandas multiprocessing apply
  13. Why does Python’s multiprocessing module import __main__ when starting a new process on Windows?
  14. Multiprocessing.Pool makes Numpy matrix multiplication slower
  15. Can I use a multiprocessing Queue in a function called by Pool.imap?
  16. Using the multiprocessing module for cluster computing
  17. multiprocessing.Pool() slower than just using ordinary functions
  18. Accessing an attribute of a multiprocessing Proxy of a class
  19. call multiprocessing in class method Python
  20. Leveraging “Copy-on-Write” to Copy Data to Multiprocessing.Pool() Worker Processes
  21. Distributed lock manager for Python
  22. How to get the return value of a function passed to multiprocessing.Process?
  23. How to share numpy random state of a parent process with child processes?
  24. multiprocessing vs multithreading vs asyncio
  25. How to let Pool.map take a lambda function
  26. Python Multiprocessing error: AttributeError: module ‘__main__’ has no attribute ‘__spec__’
  27. Python multiprocessing – tracking the process of pool.map operation
  28. Creating and updating nested dictionaries and lists inside multiprocessing.Manager object
  29. Why multiprocessing.Process behave differently on windows and linux for global object and function arguments
  30. Are locks unnecessary in multi-threaded Python code because of the GIL?

Leave a Comment