Using multiprocessing.Manager.list instead of a real list makes the calculation take ages

by

Linux uses copy-on-write when subprocesses are os.forked. To demonstrate:

import multiprocessing as mp
import numpy as np
import logging
import os

logger = mp.log_to_stderr(logging.WARNING)

def free_memory():
    total = 0
    with open('/proc/meminfo', 'r') as f:
        for line in f:
            line = line.strip()
            if any(line.startswith(field) for field in ('MemFree', 'Buffers', 'Cached')):
                field, amount, unit = line.split()
                amount = int(amount)
                if unit != 'kB':
                    raise ValueError(
                        'Unknown unit {u!r} in /proc/meminfo'.format(u = unit))
                total += amount
    return total

def worker(i):
    x = data[i,:].sum()    # Exercise access to data
    logger.warn('Free memory: {m}'.format(m = free_memory()))

def main():
    procs = [mp.Process(target = worker, args = (i, )) for i in range(4)]
    for proc in procs:
        proc.start()
    for proc in procs:
        proc.join()

logger.warn('Initial free: {m}'.format(m = free_memory()))
N = 15000
data = np.ones((N,N))
logger.warn('After allocating data: {m}'.format(m = free_memory()))

if __name__ == '__main__':
    main()

which yielded

[WARNING/MainProcess] Initial free: 2522340
[WARNING/MainProcess] After allocating data: 763248
[WARNING/Process-1] Free memory: 760852
[WARNING/Process-2] Free memory: 757652
[WARNING/Process-3] Free memory: 757264
[WARNING/Process-4] Free memory: 756760

This shows that initially there was roughly 2.5GB of free memory.
After allocating a 15000×15000 array of float64s, there was 763248 KB free. This roughly makes sense since 15000**2*8 bytes = 1.8GB and the drop in memory, 2.5GB – 0.763248GB is also roughly 1.8GB.

Now after each process is spawned, the free memory is again reported to be ~750MB. There is no significant decrease in free memory, so I conclude the system must be using copy-on-write.

Conclusion: If you do not need to modify the data, defining it at the global level of the __main__ module is a convenient and (at least on Linux) memory-friendly way to share it among subprocesses.

More Related Contents:

  1. How to run functions in parallel?
  2. multiprocessing.Pool: When to use apply, apply_async or map?
  3. Shared-memory objects in multiprocessing
  4. Shared memory in multiprocessing
  5. Dead simple example of using Multiprocessing Queue, Pool and Locking
  6. Share Large, Read-Only Numpy Array Between Multiprocessing Processes
  7. multiprocessing.Pool: What’s the difference between map_async and imap?
  8. Python Multiprocessing: Handling Child Errors in Parent
  9. Processing single file from multiple processes
  10. How to use a multiprocessing.Manager()?
  11. Solving embarassingly parallel problems using Python multiprocessing
  12. Log output of multiprocessing.Process
  13. multiprocessing vs multithreading vs asyncio in Python 3
  14. pandas multiprocessing apply
  15. Why does Python’s multiprocessing module import __main__ when starting a new process on Windows?
  16. Appending to the same list from different processes using multiprocessing
  17. Exception thrown in multiprocessing Pool not detected
  18. Using the multiprocessing module for cluster computing
  19. multiprocessing.Pool() slower than just using ordinary functions
  20. How do you pass a Queue reference to a function managed by pool.map_async()?
  21. What kind of problems (if any) would there be combining asyncio with multiprocessing?
  22. What is the difference between multiprocessing and subprocess?
  23. Python Multiprocessing Lib Error (AttributeError: __exit__)
  24. How to let Pool.map take a lambda function
  25. Show the progress of a Python multiprocessing pool imap_unordered call?
  26. Python Multiprocessing error: AttributeError: module ‘__main__’ has no attribute ‘__spec__’
  27. Multiprocessing AsyncResult.get() hangs in Python 3.7.2 but not in 3.6
  28. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  29. How to signal slots in a GUI from a different process?
  30. Terminate a Python multiprocessing program once a one of its workers meets a certain condition

Leave a Comment