Share a list between different processes?

by

One way is to use a manager object and create your shared list object from it:

from multiprocessing import Manager, Pool

input_list = ['A', 'B', 'C', 'D', 'E', 'F']

manager = Manager()
shared_list = manager.list()

def do_stuff(element):
    element_dict = {}
    element_dict['name'] = element
    shared_list.append(element_dict)
    if len(shared_list) > 3:
        print('list > 3')

pool = Pool(processes=6)
pool.map(do_stuff, input_list)
pool.close()

Remember, unlike threads, processes do not share memory space. (When spawned, each process gets its own copy of the memory footprint of the spawning process, and then runs with it.) So they can only communicate via some form of IPC (interprocess communication). In Python, one such method is multiprocessing.Manager and the data structures it exposes, e.g. list or dict. These are used in code as easily as their built-in equivalents, but under the hood utilize some form of IPC (sockets probably).

Edit Feb 1, 2022: Removed unneeded
global shared_list declaration from the function, since the object is not being replaced.

More Related Contents:

  1. Appending to the same list from different processes using multiprocessing
  2. What does += mean in this context?
  3. How to name a file using a value in a list in a dictionary?
  4. What does “list comprehension” and similar mean? How does it work and how can I use it?
  5. Why does append() always return None in Python? [duplicate]
  6. Convert all strings in a list to int
  7. Get unique values from a list in python [duplicate]
  8. Element-wise addition of 2 lists?
  9. Sum a list of numbers in Python
  10. multiprocessing: Understanding logic behind `chunksize`
  11. List comprehension vs generator expression’s weird timeit results?
  12. Modify a list while iterating
  13. How do I create a list with numbers between two values?
  14. How do I loop through a list by twos? [duplicate]
  15. converting a list of integers into range in python
  16. List assignment with [:]
  17. Multiprocessing working in Python but not in iPython
  18. How to make a set of lists
  19. What’s the difference between lists enclosed by square brackets and parentheses in Python?
  20. Difference between using [] and list() in Python
  21. TypeError: ‘float’ object is not subscriptable
  22. Python .sort() not working as expected
  23. Fastest way to count number of occurrences in a Python list
  24. how to to terminate process using python’s multiprocessing
  25. Creating a dictionary with list of lists in Python
  26. Python: Elegant and efficient ways to mask a list
  27. How to utilize all cores with python multiprocessing
  28. How can I share a class between processes?
  29. list comprehension with multiple conditions (python)
  30. How to convert only numbers in a mixed list into float

Leave a Comment