multiprocessing pool example does not work and freeze the kernel

by

This happens because you didn’t protect your “procedural” part of the code from re-execution when your child processes are importing f.

They need to import f, because Windows doesn’t support forking as start method for new processes (only spawn). A new Python process has to be started from scratch, f imported and this import will also trigger another Pool to be created in all child-processes … and their child-processes and their child-processes…

To prevent this recursion, you have to insert an if __name__ == '__main__': -line between the upper part, which should run on imports and a lower part, which should only run when your script is executed as the main script (only the case for the parent).

from multiprocessing import Pool

def f(x):
  return x*x

if __name__ == '__main__': # protect your program's entry point

    p = Pool(6)
    print(p.map(f, range(10)))

Separating your code like that is mandatory for multiprocessing on Windows and Unix-y systems when used with ‘spawn’ or ‘forkserver’ start-method instead of default ‘fork’. In general, start-methods can be modified with multiprocessing.set_start_method(method).

Since Python 3.8, macOS also uses ‘spawn’ instead of ‘fork’ as default.

It’s general a good practice to separate any script in upper “definition” and lower “execution as main”, to make code importable without unnecessary executions of parts only relevant when run as top level script. Last but not least, it facilitates understanding the control flow of your program when you don’t intermix definitions and executions.

More Related Contents:

  1. Sharing a result queue among several processes
  2. multiprocessing: Understanding logic behind `chunksize`
  3. Python multiprocessing.Pool: AttributeError
  4. Processes stuck in loop with PyInstaller-executable
  5. How to use multiprocessing pool.map with multiple arguments
  6. Python multiprocessing PicklingError: Can’t pickle
  7. RuntimeError on windows trying python multiprocessing
  8. What are the differences between the threading and multiprocessing modules?
  9. multiprocessing.dummy in Python is not utilising 100% cpu
  10. Script using multiprocessing module does not terminate
  11. Compulsory usage of if __name__==”__main__” in windows while using multiprocessing [duplicate]
  12. Same output in different workers in multiprocessing
  13. Poor scaling of multiprocessing Pool.map() on a list of large objects: How to achieve better parallel scaling in python?
  14. multiprocessing.pool.MaybeEncodingError: ‘TypeError(“cannot serialize ‘_io.BufferedReader’ object”,)’
  15. Leveraging “Copy-on-Write” to Copy Data to Multiprocessing.Pool() Worker Processes
  16. Multiprocessing launching too many instances of Python VM
  17. Modify object in python multiprocessing
  18. How to get the return value of a function passed to multiprocessing.Process?
  19. What’s the difference between ThreadPool vs Pool in the multiprocessing module?
  20. how to to terminate process using python’s multiprocessing
  21. Python Multiprocessing error: AttributeError: module ‘__main__’ has no attribute ‘__spec__’
  22. Multiprocessing AsyncResult.get() hangs in Python 3.7.2 but not in 3.6
  23. Creating and updating nested dictionaries and lists inside multiprocessing.Manager object
  24. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  25. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  26. How can I abort a task in a multiprocessing.Pool after a timeout?
  27. How can I share a class between processes?
  28. How to limit number of CPU’s used by a python script w/o terminal or multiprocessing library?
  29. Python multiprocessing linux windows difference
  30. multiprocessing: How can I ʀᴇʟɪᴀʙʟʏ redirect stdout from a child process?

Leave a Comment