“chunksize” parameter in multiprocessing.Pool.map

by

Looking at the documentation for Pool.map it seems you’re almost correct: the chunksize parameter will cause the iterable to be split into pieces of approximately that size, and each piece is submitted as a separate task.

So in your example, yes, map will take the first 10 (approximately), submit it as a task for a single processor… then the next 10 will be submitted as another task, and so on. Note that it doesn’t mean that this will make the processors alternate every 10 files, it’s quite possible that processor #1 ends up getting 1-10 AND 11-20, and processor #2 gets 21-30 and 31-40.

More Related Contents:

  1. Multiprocessing vs Threading Python [duplicate]
  2. RuntimeError on windows trying python multiprocessing
  3. multiprocessing: How do I share a dict among multiple processes?
  4. multiprocessing: sharing a large read-only object between processes?
  5. Python sharing a lock between processes
  6. How to use multiprocessing queue in Python?
  7. where to put freeze_support() in a Python script?
  8. Multiprocessing a for loop?
  9. Using Python’s Multiprocessing module to execute simultaneous and separate SEAWAT/MODFLOW model runs
  10. Poor scaling of multiprocessing Pool.map() on a list of large objects: How to achieve better parallel scaling in python?
  11. Concurrent.futures vs Multiprocessing in Python 3
  12. Python Multiprocessing.Pool lazy iteration
  13. Share list between process in python server
  14. High Memory Usage Using Python Multiprocessing
  15. Python multiprocessing.Pool: AttributeError
  16. Python: Getting a traceback from a multiprocessing.Process
  17. Connection problems with SQLAlchemy and multiple processes
  18. OSError 38 [Errno 38] with multiprocessing
  19. Can multiprocessing Process class be run from IDLE
  20. Python multiprocess profiling
  21. What’s the difference between ThreadPool vs Pool in the multiprocessing module?
  22. Globals variables and Python multiprocessing [duplicate]
  23. Python OpenCV multiprocessing cv2.VideoCapture mp4
  24. Using 100% of all cores with the multiprocessing module
  25. Mix Python Twisted with multiprocessing?
  26. Python multiprocessing memory usage
  27. How to limit number of CPU’s used by a python script w/o terminal or multiprocessing library?
  28. Multiprocessing pool with an iterator
  29. multiprocessing: How can I ʀᴇʟɪᴀʙʟʏ redirect stdout from a child process?
  30. Sharing a lock between gunicorn workers

Leave a Comment