Compulsory usage of if __name__==”__main__” in windows while using multiprocessing [duplicate]

by

Expanding a bit on the good answer you already got, it helps if you understand what Linux-y systems do. They spawn new processes using fork(), which has two good consequences:

  1. All data structures existing in the main program are visible to the child processes. They actually work on copies of the data.
  2. The child processes start executing at the instruction immediately following the fork() in the main program – so any module-level code already executed in the module will not be executed again.

fork() isn’t possible in Windows, so on Windows each module is imported anew by each child process. So:

  1. On Windows, no data structures existing in the main program are visible to the child processes; and,
  2. All module-level code is executed in each child process.

So you need to think a bit about which code you want executed only in the main program. The most obvious example is that you want code that creates child processes to run only in the main program – so that should be protected by __name__ == '__main__'. For a subtler example, consider code that builds a gigantic list, which you intend to pass out to worker processes to crawl over. You probably want to protect that too, because there’s no point in this case to make each worker process waste RAM and time building their own useless copies of the gigantic list.

Note that it’s a Good Idea to use __name__ == "__main__" appropriately even on Linux-y systems, because it makes the intended division of work clearer. Parallel programs can be confusing – every little bit helps 😉

More Related Contents:

  1. RuntimeError on windows trying python multiprocessing
  2. Why does Python’s multiprocessing module import __main__ when starting a new process on Windows?
  3. Multiprocessing launching too many instances of Python VM
  4. multiprocessing pool example does not work and freeze the kernel
  5. Python multiprocessing linux windows difference
  6. multiprocessing: How can I ʀᴇʟɪᴀʙʟʏ redirect stdout from a child process?
  7. How do I hide the console when I use os.system() or subprocess.call()?
  8. How to run functions in parallel?
  9. How do you get a directory listing sorted by creation date in python?
  10. How do I read text from the clipboard?
  11. Running Python on Windows for Node.js dependencies
  12. multiprocessing.Pool: What’s the difference between map_async and imap?
  13. How to add a custom CA Root certificate to the CA Store used by pip in Windows?
  14. How to find the real user home directory using python?
  15. Processing single file from multiple processes
  16. Solving embarassingly parallel problems using Python multiprocessing
  17. multiprocessing vs multithreading vs asyncio in Python 3
  18. What’s the best way to duplicate fork() in windows?
  19. Exception thrown in multiprocessing Pool not detected
  20. Python: Start new command prompt on Windows and wait for it finish/exit
  21. python: windows equivalent of SIGALRM
  22. Python Pip install Error: Unable to find vcvarsall.bat. Tried all solutions [duplicate]
  23. How do you pass a Queue reference to a function managed by pool.map_async()?
  24. What kind of problems (if any) would there be combining asyncio with multiprocessing?
  25. Run multiple commands in different SSH servers in parallel using Python Paramiko
  26. PyAudio failed to install on Windows 10
  27. Show the progress of a Python multiprocessing pool imap_unordered call?
  28. Silent printing of a PDF in Python
  29. Export Charts from Excel as images using Python
  30. Sleep for exact time in python

Leave a Comment