How to use a multiprocessing.Manager()?

by

Manager proxy objects are unable to propagate changes made to (unmanaged) mutable objects inside a container. So in other words, if you have a manager.list() object, any changes to the managed list itself are propagated to all the other processes. But if you have a normal Python list inside that list, any changes to the inner list are not propagated, because the manager has no way of detecting the change.

In order to propagate the changes, you have to use manager.list() objects for the nested lists too (requires Python 3.6 or newer), or you need to modify the manager.list() object directly (see the note on manager.list in Python 3.5 or older).

For example, consider the following code and its output:

import multiprocessing
import time

def f(ns, ls, di):
    ns.x += 1
    ns.y[0] += 1
    ns_z = ns.z
    ns_z[0] += 1
    ns.z = ns_z

    ls[0] += 1
    ls[1][0] += 1 # unmanaged, not assigned back
    ls_2 = ls[2]  # unmanaged...
    ls_2[0] += 1
    ls[2] = ls_2  # ... but assigned back
    ls[3][0] += 1 # managed, direct manipulation

    di[0] += 1
    di[1][0] += 1 # unmanaged, not assigned back
    di_2 = di[2]  # unmanaged...
    di_2[0] += 1
    di[2] = di_2  # ... but assigned back
    di[3][0] += 1 # managed, direct manipulation

if __name__ == '__main__':
    manager = multiprocessing.Manager()
    ns = manager.Namespace()
    ns.x = 1
    ns.y = [1]
    ns.z = [1]
    ls = manager.list([1, [1], [1], manager.list([1])])
    di = manager.dict({0: 1, 1: [1], 2: [1], 3: manager.list([1])})

    print('before', ns, ls, ls[2], di, di[2], sep='\n')
    p = multiprocessing.Process(target=f, args=(ns, ls, di))
    p.start()
    p.join()
    print('after', ns, ls, ls[2], di, di[2], sep='\n')

Output:

before
Namespace(x=1, y=[1], z=[1])
[1, [1], [1], <ListProxy object, typeid 'list' at 0x10b8c4630>]
[1]
{0: 1, 1: [1], 2: [1], 3: <ListProxy object, typeid 'list' at 0x10b8c4978>}
[1]
after
Namespace(x=2, y=[1], z=[2])
[2, [1], [2], <ListProxy object, typeid 'list' at 0x10b8c4630>]
[2]
{0: 2, 1: [1], 2: [2], 3: <ListProxy object, typeid 'list' at 0x10b8c4978>}
[2]

As you can see, when a new value is assigned directly to the managed container, it changes; when it is assigned to a mutable container within the managed container, it doesn’t change; but if the mutable container is then reassigned to the managed container, it changes again. Using a nested managed container also works, detecting changes directly without having to assign back to the parent container.

More Related Contents:

  1. How can I force division to be floating point? Division keeps rounding down to 0?
  2. Why does the division get rounded to an integer?
  3. How to use multiprocessing pool.map with multiple arguments
  4. Can’t pickle when using multiprocessing Pool.map()
  5. How can I read large text files line by line, without loading it into memory?
  6. How do I keep Python print from adding newlines or spaces? [duplicate]
  7. How to get string objects instead of Unicode from JSON?
  8. What are the differences between the threading and multiprocessing modules?
  9. Python unexpected EOF while parsing
  10. How to print variables without spaces between values [duplicate]
  11. Why does Python print unicode characters when the default encoding is ASCII?
  12. Combine Pool.map with shared memory Array in Python multiprocessing
  13. Sharing a complex object between processes?
  14. Python 2 CSV writer produces wrong line terminator on Windows
  15. Printing without newline (print ‘a’,) prints a space, how to remove?
  16. Python command line input in a process
  17. yet another confusion with multiprocessing error, ‘module’ object has no attribute ‘f’
  18. Multiprocessing Bomb
  19. compare lines in a file with a string
  20. Multiprocessing launching too many instances of Python VM
  21. Replace list of list with “condensed” list of list while maintaining order
  22. Modify object in python multiprocessing
  23. What is more ‘pythonic’ for ‘not’ [duplicate]
  24. Python Multiprocess Pool. How to exit the script when one of the worker process determines no more work needs to be done?
  25. Printing a string prints ‘u’ before the string in Python?
  26. multiprocessing pool example does not work and freeze the kernel
  27. How do implicit relative imports work in Python?
  28. Using multiprocessing.Process with a maximum number of simultaneous processes
  29. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  30. Python: How to resize an image using PIL module

Leave a Comment