Python – appending to same file from multiple threads

by

The solution is to write to the file in one thread only.

import Queue  # or queue in Python 3
import threading

class PrintThread(threading.Thread):
    def __init__(self, queue):
        threading.Thread.__init__(self)
        self.queue = queue

    def printfiles(self, p):
        for path, dirs, files in os.walk(p):
            for f in files:
                print(f, file=output)

    def run(self):
        while True:
            result = self.queue.get()
            self.printfiles(result)
            self.queue.task_done()

class ProcessThread(threading.Thread):
    def __init__(self, in_queue, out_queue):
        threading.Thread.__init__(self)
        self.in_queue = in_queue
        self.out_queue = out_queue

    def run(self):
        while True:
            path = self.in_queue.get()
            result = self.process(path)
            self.out_queue.put(result)
            self.in_queue.task_done()

    def process(self, path):
        # Do the processing job here

pathqueue = Queue.Queue()
resultqueue = Queue.Queue()
paths = getThisFromSomeWhere()

output = codecs.open('file', 'a')

# spawn threads to process
for i in range(0, 5):
    t = ProcessThread(pathqueue, resultqueue)
    t.setDaemon(True)
    t.start()

# spawn threads to print
t = PrintThread(resultqueue)
t.setDaemon(True)
t.start()

# add paths to queue
for path in paths:
    pathqueue.put(path)

# wait for queue to get empty
pathqueue.join()
resultqueue.join()

More Related Contents:

  1. How can I use threading in Python?
  2. Timeout on a function call
  3. threading.Timer – repeat function every ‘n’ seconds
  4. Are lists thread-safe?
  5. Freezing/Hanging tkinter GUI in waiting for the thread to complete
  6. Daemon Threads Explanation
  7. What is the use of join() in Python threading?
  8. thread starts running before calling Thread.start
  9. How to terminate a thread when main program ends?
  10. A very simple multithreading parallel URL fetching (without queue)
  11. The right way to limit maximum number of threads running at once?
  12. Thread vs. Threading
  13. How to pause and resume a thread using the threading module?
  14. Meaning of daemon property on Python Threads
  15. How to use threading in Python?
  16. What is the use of join() in threading?
  17. Return value from thread
  18. How to start threads at the same time in Python [duplicate]
  19. Restarting a thread in Python
  20. How to obtain a Thread id in Python?
  21. How do I use threading in Python?
  22. With “args” vs without “args” to pass arguments to a thread in Python
  23. Does Python support multithreading? Can it speed up execution time?
  24. Is there any way to kill a Thread?
  25. background function in Python
  26. Python: Something like `map` that works on threads [closed]
  27. python threading blocks
  28. TypeError in Threading. function takes x positional argument but y were given
  29. Execute Python function in Main thread from call in Dummy thread
  30. Why does Paramiko hang if you use it while loading a module?

Leave a Comment