boost.python not supporting parallelism?

What you are running into is the python Global Interpreter Lock. The GIL only allows one thread at a time to run in the python interpreter.

One of the advantages of Boost.Python is that you can release the GIL, do C++ stuff, and then take it back when you are done. This is also a responsibility however. Python normally releases the GIL at regular intervals, to give other threads a chance to run. If you are in C++, this is your job. If you go crunch numbers for 2 hours while holding the GIL, you will freeze the whole interpreter.

This can be easy to fix with a little reverse RAII:

class releaseGIL{
public:
    inline releaseGIL(){
        save_state = PyEval_SaveThread();
    }

    inline ~releaseGIL(){
        PyEval_RestoreThread(save_state);
    }
private:
    PyThreadState *save_state;
};

Now you can change your code like so:

class Foo{
public:
    Foo(){}
    void run(){
        {
            releaseGIL unlock = releaseGIL();
            int seconds = 2;
            clock_t endwait;
            endwait = clock () + seconds * CLOCKS_PER_SEC ;
            while (clock() < endwait) {}
        }
    }   
};

It is VERY important to note that you MUST NOT touch any python code, or python data or call in to the interpreter while not holding the GIL. This will cause your interpreter to crash.

It is also possible to go the other way. A thread not currently holding the GIL can acquire it, and make calls in to python. This can be a thread that released the GIL earlier, or one that started in c++ and never had the GIL. Here is the RAII class for that:

class AcquireGIL 
{
public:
    inline AcquireGIL(){
        state = PyGILState_Ensure();
    }

    inline ~AcquireGIL(){
        PyGILState_Release(state);
    }
private:
    PyGILState_STATE state;
};

Usage is left as an exercise for the student.

Additional note (I always forget to mention this):

If you are going to be messing with the GIL in c++ your module definition needs to start with this code:

BOOST_PYTHON_MODULE(ModuleName)
{
    PyEval_InitThreads();

    ...
}

Leave a Comment