Threading in Python [closed]

by

In order of increasing complexity:

Use the threading module

Pros:

  • It’s really easy to run any function (any callable in fact) in its
    own thread.
  • Sharing data is if not easy (locks are never easy :), at
    least simple.

Cons:

  • As mentioned by Juergen Python threads cannot actually concurrently access state in the interpreter (there’s one big lock, the infamous Global Interpreter Lock.) What that means in practice is that threads are useful for I/O bound tasks (networking, writing to disk, and so on), but not at all useful for doing concurrent computation.

Use the multiprocessing module

In the simple use case this looks exactly like using threading except each task is run in its own process not its own thread. (Almost literally: If you take Eli’s example, and replace threading with multiprocessing, Thread, with Process, and Queue (the module) with multiprocessing.Queue, it should run just fine.)

Pros:

  • Actual concurrency for all tasks (no Global Interpreter Lock).
  • Scales to multiple processors, can even scale to multiple machines.

Cons:

  • Processes are slower than threads.
  • Data sharing between processes is trickier than with threads.
  • Memory is not implicitly shared. You either have to explicitly share it or you have to pickle variables and send them back and forth. This is safer, but harder. (If it matters increasingly the Python developers seem to be pushing people in this direction.)

Use an event model, such as Twisted

Pros:

  • You get extremely fine control over priority, over what executes when.

Cons:

  • Even with a good library, asynchronous programming is usually harder than threaded programming, hard both in terms of understanding what’s supposed to happen and in terms of debugging what actually is happening.

In all cases I’m assuming you already understand many of the issues involved with multitasking, specifically the tricky issue of how to share data between tasks. If for some reason you don’t know when and how to use locks and conditions you have to start with those. Multitasking code is full of subtleties and gotchas, and it’s really best to have a good understanding of concepts before you start.

More Related Contents:

  1. Python Global Interpreter Lock (GIL) workaround on multi-core systems using taskset on Linux?
  2. How to use asyncio with threading in Python is Method threading.Timer
  3. Multiprocessing vs Threading Python [duplicate]
  4. Are lists thread-safe?
  5. Catch a thread’s exception in the caller thread?
  6. python multithreading wait till all threads finished
  7. thread starts running before calling Thread.start
  8. How to use multiprocessing queue in Python?
  9. How do you stop numpy from multithreading? [duplicate]
  10. How to combine python asyncio with threads?
  11. Start a flask application in separate thread
  12. is python capable of running on multiple cores?
  13. Can you perform multi-threaded tasks within Django?
  14. Why is a Python I/O bound task not blocked by the GIL?
  15. Thread vs. Threading
  16. Most Pythonic way to kill a thread after some period of time
  17. How to use threading in Python?
  18. In PyQt, what is the best way to share data between the main window and a thread
  19. Running a Tkinter form in a separate thread
  20. Python creating a shared variable between threads
  21. error: can’t start new thread
  22. Deadlock with logging multiprocess/multithread python script
  23. How to start threads at the same time in Python [duplicate]
  24. Is a variable swap guaranteed to be atomic in python?
  25. Restarting a thread in Python
  26. How to obtain a Thread id in Python?
  27. Dump stacktraces of all active Threads
  28. How to use QThread correctly in pyqt with moveToThread()?
  29. ‘sys.excepthook’ and threading
  30. Are locks unnecessary in multi-threaded Python code because of the GIL?

Leave a Comment