Embedding python in multithreaded C application

by

I had exactly the same problem and it is now solved by using PyEval_SaveThread() immediately after PyEval_InitThreads(), as you suggest above. However, my actual problem was that I used PyEval_InitThreads() after PyInitialise() which then caused PyGILState_Ensure() to block when called from different, subsequent native threads. In summary, this is what I do now:

  1. There is global variable: 

    static int gil_init = 0;

  2. From a main thread load the native C extension and start the Python interpreter:

    Py_Initialize()

  3. From multiple other threads my app concurrently makes a lot of calls into the Python/C API:

    if (!gil_init) {
    gil_init = 1;
    PyEval_InitThreads();
    PyEval_SaveThread();
}
state = PyGILState_Ensure();
// Call Python/C API functions...    
PyGILState_Release(state);

  4. From the main thread stop the Python interpreter

    Py_Finalize()

All other solutions I’ve tried either caused random Python sigfaults or deadlock/blocking using PyGILState_Ensure().

The Python documentation really should be more clear on this and at least provide an example for both the embedding and extension use cases.

More Related Contents:

  1. Create an object using Python’s C API
  2. Python Global Interpreter Lock (GIL) workaround on multi-core systems using taskset on Linux?
  3. Multithreading with Python and C api
  4. What is the global interpreter lock (GIL) in CPython?
  5. threading.Timer – repeat function every ‘n’ seconds
  6. Freezing/Hanging tkinter GUI in waiting for the thread to complete
  7. Does Python support multithreading? Can it speed up execution time?
  8. What is the use of join() in Python threading?
  9. Is there any way to kill a Thread?
  10. Speed comparison with Project Euler: C vs Python vs Erlang vs Haskell
  11. How to start and stop a thread
  12. It is more efficient to use if-return-return or if-else-return?
  13. background function in Python
  14. Python: Something like `map` that works on threads [closed]
  15. Multi-threaded use of SQLAlchemy
  16. Python – appending to same file from multiple threads
  17. deciding among subprocess, multiprocessing, and thread in Python?
  18. Accessing OpenCV CUDA Functions from Python (No PyCUDA)
  19. How to get a faster speed when using multi-threading in python
  20. How to run multiple functions at the same time?
  21. I have Python on my Ubuntu system, but gcc can’t find Python.h
  22. What is the use of join() in threading?
  23. Python: How to NOT wait for a thread to finish to carry on? [duplicate]
  24. Where can I find source or algorithm of Python’s hash() function?
  25. Threads vs. Async
  26. multiprocess or threading in python?
  27. ImportError: dynamic module does not define init function (initfizzbuzz)
  28. Is it thread-safe when using tf.Session in inference service?
  29. SWIG interfacing C library to Python (Creating ‘iterable’ Python data type from C ‘sequence’ struct)
  30. Are locks unnecessary in multi-threaded Python code because of the GIL?

Leave a Comment