Python: Writing to a single file with queue while using multiprocessing Pool

by

Multiprocessing pools implement a queue for you. Just use a pool method that returns the worker return value to the caller. imap works well:

import multiprocessing 
import re

def mp_worker(filename):
    with open(filename) as f:
        text = f.read()
    m = re.findall("x+", text)
    count = len(max(m, key=len))
    return filename, count

def mp_handler():
    p = multiprocessing.Pool(32)
    with open('infilenamess.txt') as f:
        filenames = [line for line in (l.strip() for l in f) if line]
    with open('results.txt', 'w') as f:
        for result in p.imap(mp_worker, filenames):
            # (filename, count) tuples from worker
            f.write('%s: %d\n' % result)

if __name__=='__main__':
    mp_handler()

More Related Contents:

  1. Filling a queue and managing multiprocessing in python
  2. Can I use a multiprocessing Queue in a function called by Pool.imap?
  3. How do you pass a Queue reference to a function managed by pool.map_async()?
  4. Can’t pickle when using multiprocessing Pool.map()
  5. Sharing a result queue among several processes
  6. Keyboard Interrupts with python’s multiprocessing Pool
  7. Python Process Pool non-daemonic?
  8. Combine Pool.map with shared memory Array in Python multiprocessing
  9. Multiprocessing.Pool makes Numpy matrix multiplication slower
  10. Multiprocessing – Pipe vs Queue
  11. multiprocessing.Pool() slower than just using ordinary functions
  12. Passing multiple parameters to pool.map() function in Python [duplicate]
  13. Python Multiprocess Pool. How to exit the script when one of the worker process determines no more work needs to be done?
  14. How to let Pool.map take a lambda function
  15. python multiprocessing: some functions do not return when they are complete (queue material too big)
  16. Python multiprocessing – tracking the process of pool.map operation
  17. How to use Python multiprocessing Pool.map to fill numpy array in a for loop
  18. multiprocessing.Pool: When to use apply, apply_async or map?
  19. Shared variable in python’s multiprocessing
  20. Share Large, Read-Only Numpy Array Between Multiprocessing Processes
  21. Python Multiprocessing: Handling Child Errors in Parent
  22. Processing single file from multiple processes
  23. How to use a multiprocessing.Manager()?
  24. Appending to the same list from different processes using multiprocessing
  25. Exception thrown in multiprocessing Pool not detected
  26. What kind of problems (if any) would there be combining asyncio with multiprocessing?
  27. What is the difference between multiprocessing and subprocess?
  28. Python Multiprocessing Lib Error (AttributeError: __exit__)
  29. Show the progress of a Python multiprocessing pool imap_unordered call?
  30. Easiest way to read csv files with multiprocessing in Pandas

Leave a Comment