Python multiprocessing memory usage

by

The multiprocessing module is effectively based on the fork system call which creates a copy of the current process. Since you are loading the huge data before you fork (or create the multiprocessing.Process), the child process inherits a copy of the data.

However, if the operating system you are running on implements COW (copy-on-write), there will only actually be one copy of the data in physical memory unless you modify the data in either the parent or child process (both parent and child will share the same physical memory pages, albeit in different virtual address spaces); and even then, additional memory will only be allocated for the changes (in pagesize increments).

You can avoid this situation by calling multiprocessing.Process before you load your huge data. Then the additional memory allocations will not be reflected in the child process when you load the data in the parent.

Edit: reflecting @Janne Karila’s comment in the answer, as it is so relevant: “Note also that every Python object contains a reference count that is modified whenever the object is accessed. So, just reading a data structure can cause COW to copy.”

More Related Contents:

  1. Why does multiprocessing use only a single core after I import numpy?
  2. Leveraging “Copy-on-Write” to Copy Data to Multiprocessing.Pool() Worker Processes
  3. OSError 38 [Errno 38] with multiprocessing
  4. Python multiprocessing linux windows difference
  5. How do I determine the size of an object in Python?
  6. How to use `subprocess` command with pipes
  7. RuntimeError on windows trying python multiprocessing
  8. multiprocessing global variable updates not returned to parent
  9. How should I log while using multiprocessing in Python?
  10. Is it possible to change the Environment of a parent process in Python?
  11. Python multiprocessing safely writing to a file
  12. When are objects garbage collected in python?
  13. Python FTP get the most recent file by date
  14. Python: how to kill child process(es) when parent dies?
  15. Access Denied page with headless Chrome on Linux while headed Chrome works on windows using Selenium through Python
  16. Compulsory usage of if __name__==”__main__” in windows while using multiprocessing [duplicate]
  17. Linux command-line call not returning what it should from os.system?
  18. Simulate keystroke in Linux with Python
  19. Concurrent.futures vs Multiprocessing in Python 3
  20. Parallelizing a Numpy vector operation
  21. “chunksize” parameter in multiprocessing.Pool.map
  22. Connection problems with SQLAlchemy and multiple processes
  23. How to split a huge csv file based on content of first column?
  24. Tkinter.PhotoImage doesn’t not support png image
  25. Python multiprocess profiling
  26. What’s the difference between ThreadPool vs Pool in the multiprocessing module?
  27. How can I send a signal from a python program?
  28. How to make python window run as “Always On Top”?
  29. Get local network interface addresses using only proc?
  30. Mix Python Twisted with multiprocessing?

Leave a Comment