Why multiprocessing.Process behave differently on windows and linux for global object and function arguments

by

On Linux (and other Unix-like OSs), Python’s multiprocessing module using fork() to create new child processes that efficiently inherit a copy of the parent process’s memory state. That means the interpreter doesn’t need to pickle the objects that are being passed as the Process‘s args since the child process will already have them available in their normal form.

Windows doesn’t have a fork() system call however, so the multiprocessing module needs to do a bit more work to make the child-spawning process work. The fork()-based implementation came first, and the non-forking Windows implementation came later.

It’s worth noting that the Python developers had often felt it was a bit of a misfeature for the creation of child processes to differ so much based on the platform you’re running Python on. So in Python 3.4, a new system was added to allow you to select the start method that you would prefer to use. The options are "fork", "forkserver" and "spawn". The "fork" method remains the default on Unix-like systems (where it was the only implementation in earlier versions of Python). The "spawn" method is the default (and only) option on Windows, but now can be used on Unix-like systems too. The "forkserver" method is sort of a hybrid between the two (and only available on some Unix-like systems). You can read more about the differences between the methods in the documentation.

More Related Contents:

  1. Python multiprocessing PicklingError: Can’t pickle
  2. Why does multiprocessing use only a single core after I import numpy?
  3. Sharing a result queue among several processes
  4. Keyboard Interrupts with python’s multiprocessing Pool
  5. multiprocessing: Understanding logic behind `chunksize`
  6. Multiprocessing: How to use Pool.map on a function defined in a class?
  7. How to increment a shared counter from multiple processes?
  8. multiprocessing.dummy in Python is not utilising 100% cpu
  9. multiprocessing in python – sharing large object (e.g. pandas dataframe) between multiple processes
  10. Same output in different workers in multiprocessing
  11. Multiprocessing.Pool makes Numpy matrix multiplication slower
  12. Can I use a multiprocessing Queue in a function called by Pool.imap?
  13. Multiprocessing working in Python but not in iPython
  14. Accessing an attribute of a multiprocessing Proxy of a class
  15. call multiprocessing in class method Python
  16. Leveraging “Copy-on-Write” to Copy Data to Multiprocessing.Pool() Worker Processes
  17. How to get the return value of a function passed to multiprocessing.Process?
  18. Python selenium multiprocessing
  19. How to share numpy random state of a parent process with child processes?
  20. multiprocessing.Pool – PicklingError: Can’t pickle : attribute lookup thread.lock failed
  21. multiprocessing vs multithreading vs asyncio
  22. Globals variables and Python multiprocessing [duplicate]
  23. Python OpenCV multiprocessing cv2.VideoCapture mp4
  24. how to to terminate process using python’s multiprocessing
  25. Using python multiprocessing Pool in the terminal and in code modules for Django or Flask
  26. Python multiprocessing – tracking the process of pool.map operation
  27. Creating and updating nested dictionaries and lists inside multiprocessing.Manager object
  28. Python multiprocessing application is getting stuck in docker container
  29. How to synchronize a python dict with multiprocessing
  30. Using python multiprocessing with different random seed for each process

Leave a Comment