Terminate multiple threads when any thread completes a task

by

There doesn’t seem to be a (simple) way to terminate a thread in Python.

Here is a simple example of running multiple HTTP requests in parallel:

import threading

def crawl():
    import urllib2
    data = urllib2.urlopen("http://www.google.com/").read()

    print "Read google.com"

threads = []

for n in range(10):
    thread = threading.Thread(target=crawl)
    thread.start()

    threads.append(thread)

# to wait until all three functions are finished

print "Waiting..."

for thread in threads:
    thread.join()

print "Complete."

With additional overhead, you can use a multi-process aproach that’s more powerful and allows you to terminate thread-like processes.

I’ve extended the example to use that. I hope this will be helpful to you:

import multiprocessing

def crawl(result_queue):
    import urllib2
    data = urllib2.urlopen("http://news.ycombinator.com/").read()

    print "Requested..."

    if "result found (for example)":
        result_queue.put("result!")

    print "Read site."

processs = []
result_queue = multiprocessing.Queue()

for n in range(4): # start 4 processes crawling for the result
    process = multiprocessing.Process(target=crawl, args=[result_queue])
    process.start()
    processs.append(process)

print "Waiting for result..."

result = result_queue.get() # waits until any of the proccess have `.put()` a result

for process in processs: # then kill them all off
    process.terminate()

print "Got result:", result

More Related Contents:

  1. Timeout on a function call
  2. Tkinter: How to use threads to preventing main event loop from “freezing”
  3. Can’t pickle when using multiprocessing Pool.map()
  4. Background thread with QThread in PyQt
  5. Run certain code every n seconds [duplicate]
  6. What are the differences between the threading and multiprocessing modules?
  7. Creating Threads in python
  8. PyQt4 Wait in thread for user input from GUI
  9. Execute a function after Flask returns response
  10. Thread that I can pause and resume?
  11. python multi-threading slower than serial?
  12. What is “thread local storage” in Python, and why do I need it?
  13. why isn’t numpy.mean multithreaded?
  14. PyQt – Modify GUI from another thread
  15. Python Threading inside a class
  16. Meaning of daemon property on Python Threads
  17. How to get the return value from a thread?
  18. Threaded Tkinter script crashes when creating the second Toplevel widget
  19. Are Generators Threadsafe?
  20. Is Keras thread safe?
  21. Python Tkinter Label redrawing every 10 seconds
  22. Python Global Interpreter Lock (GIL) workaround on multi-core systems using taskset on Linux?
  23. subprocess.wait() not waiting for Popen process to finish (when using threads)?
  24. How do I send data to a running python thread?
  25. Multithreading with Python and C api
  26. How to use threading to get user input realtime while main still running in python
  27. waiting for user input in separate thread
  28. Terminate a multi-thread python program
  29. How to close a thread from within?
  30. Embedding python in multithreaded C application

Leave a Comment