Boost.Python: Wrap functions to release the GIL

by

Exposing functors as methods is not officially supported. The supported approach would be to expose a non-member function that delegates to the member-function. However, this can result in a large amount of boilerplate code.

As best as I can tell, Boost.Python’s implementation does not explicitly preclude functors, as it allows for instances of python::object to be exposed as a method. However, Boost.Python does place some requirements on the type of object being exposed as a method:

  • The functor is CopyConstructible.
  • The functor is callable. I.e. instance o can be called o(a1, a2, a3).
  • The call signature must be available as meta-data during runtime. Boost.Python calls the boost::python::detail::get_signature() function to obtain this meta-data. The meta-data is used internally to setup proper invocation, as well as for dispatching from Python to C++.

The latter requirement is where it gets complex. For some reason that is not immediately clear to me, Boost.Python invokes get_signature() through a qualified-id, preventing argument dependent lookup. Therefore, all candidates for get_signature() must be declared before the calling template’s definition context. For example, the only overloads for get_signature() that are considered are those declared before the definition of templates that invoke it, such as class_, def(), and make_function(). To account for this behavior, when enabling a functor in Boost.Python, one must provide a get_signature() overload prior to including Boost.Python or explicitly provide a meta-sequence representing the signature to make_function().

Lets work through some examples of enabling functor support, as well as providing functors that support guards. I have opted to not use C++11 features. As such, there will be some boilerplate code that could be reduced with variadic templates. Additionally, all of the examples will use the same model that provides two non-member functions and a spam class that has two member-functions:

/// @brief Mockup class with member functions.
class spam
{
public:
  void action()
  {
    std::cout << "spam::action()"  << std::endl;
  }

  int times_two(int x)
  {
    std::cout << "spam::times_two()" << std::endl;
    return 2 * x;
  }
};

// Mockup non-member functions.
void action()
{
  std::cout << "action()"  << std::endl;
}

int times_two(int x)
{
  std::cout << "times_two()" << std::endl;
  return 2 * x;
}

Enabling boost::function

When using the preferred syntax for Boost.Function, decomposing the signature into meta-data that meets Boost.Python requirements can be done with Boost.FunctionTypes. Here is a complete example enabling boost::function functors to be exposed as a Boost.Python method:

#include <iostream>

#include <boost/function.hpp>
#include <boost/function_types/components.hpp>

namespace boost  {
namespace python {
namespace detail {
// get_signature overloads must be declared before including
// boost/python.hpp.  The declaration must be visible at the
// point of definition of various Boost.Python templates during
// the first phase of two phase lookup.  Boost.Python invokes the
// get_signature function via qualified-id, thus ADL is disabled.

/// @brief Get the signature of a boost::function.
template <typename Signature>
inline typename boost::function_types::components<Signature>::type
get_signature(boost::function<Signature>&, void* = 0)
{
  return typename boost::function_types::components<Signature>::type();
}

} // namespace detail
} // namespace python
} // namespace boost

#include <boost/python.hpp>

/// @brief Mockup class with member functions.
class spam
{
public:
  void action()
  {
    std::cout << "spam::action()"  << std::endl;
  }

  int times_two(int x)
  {
    std::cout << "spam::times_two()" << std::endl;
    return 2 * x;
  }
};

// Mockup non-member functions.
void action()
{
  std::cout << "action()"  << std::endl;
}

int times_two(int x)
{
  std::cout << "times_two()" << std::endl;
  return 2 * x;
}

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;

  // Expose class and member-function.
  python::class_<spam>("Spam")
    .def("action",  &spam::action)
    .def("times_two", boost::function<int(spam&, int)>(
        &spam::times_two))
    ;

  // Expose non-member function.
  python::def("action",  &action);
  python::def("times_two", boost::function<int()>(
      boost::bind(&times_two, 21)));
}

And its usage:

>>> import example
>>> spam = example.Spam()
>>> spam.action()
spam::action()
>>> spam.times_two(5)
spam::times_two()
10
>>> example.action()
action()
>>> example.times_two()
times_two()
42

When providing a functor that will invoke a member-function, the provided signature needs to be the non-member function equivalent. In this case, int(spam::*)(int) becomes int(spam&, int).

// ...
  .def("times_two", boost::function<int(spam&, int)>(
        &spam::times_two))
  ;

Also, arguments can be bound to the functors with boost::bind. For example, calling example.times_two() does not have to provide an argument, as 21 is already bound to the functor.

python::def("times_two", boost::function<int()>(
    boost::bind(&times_two, 21)));

Custom functor with guards

Expanding upon the above example, one can enable custom functor types to be used with Boost.Python. Lets create a functor, called guarded_function, that will use RAII, only invoking the wrapped function during the RAII object’s lifetime.

/// @brief Functor that will invoke a function while holding a guard.
///        Upon returning from the function, the guard is released.
template <typename Signature,
          typename Guard>
class guarded_function
{
public:

  typedef typename boost::function_types::result_type<Signature>::type
      result_type;

  template <typename Fn>
  guarded_function(Fn fn)
    : fn_(fn)
  {}

  result_type operator()()
  {
    Guard g;
    return fn_();
  }

  // ... overloads for operator()

private:
  boost::function<Signature> fn_;
};

The guarded_function provides similar semantics to the Python with statement. Thus, to keep with the Boost.Python API name choices, a with() C++ function will provide a way to create functors.

/// @brief Create a callable object with guards.
template <typename Guard,
          typename Fn>
boost::python::object
with(Fn fn)
{
   return boost::python::make_function(
     guarded_function<Guard, Fn>(fn), ...);
}

This allows for functions to be exposed which will run with a guard in a non-intrusive manner:

class no_gil; // Guard

// ...
  .def("times_two", with<no_gil>(&spam::times_two))
  ;

Additionally, the with() function provides the ability to deduce the function signatures, allowing the meta-data signature to be explicitly provided to Boost.Python rather than having to overload boost::python::detail::get_signature().

Here is the complete example, using two RAII types:

  • no_gil: Releases GIL in constructor, and reacquires GIL in destructor.
  • echo_guard: Prints in constructor and destructor.
#include <iostream>

#include <boost/function.hpp>
#include <boost/function_types/components.hpp>
#include <boost/function_types/function_type.hpp>
#include <boost/function_types/result_type.hpp>
#include <boost/python.hpp>
#include <boost/tuple/tuple.hpp>

namespace detail {

/// @brief Functor that will invoke a function while holding a guard.
///        Upon returning from the function, the guard is released.
template <typename Signature,
          typename Guard>
class guarded_function
{
public:

  typedef typename boost::function_types::result_type<Signature>::type
      result_type;

  template <typename Fn>
  guarded_function(Fn fn)
    : fn_(fn)
  {}

  result_type operator()()
  {
    Guard g;
    return fn_();
  }

  template <typename A1>
  result_type operator()(A1 a1)
  {
    Guard g;
    return fn_(a1);
  }

  template <typename A1, typename A2>
  result_type operator()(A1 a1, A2 a2)
  {
    Guard g;
    return fn_(a1, a2);
  }

private:
  boost::function<Signature> fn_;
};

/// @brief Provides signature type.
template <typename Signature>
struct mpl_signature
{
  typedef typename boost::function_types::components<Signature>::type type;
};

// Support boost::function.
template <typename Signature>
struct mpl_signature<boost::function<Signature> >:
  public mpl_signature<Signature>
{};

/// @brief Create a callable object with guards.
template <typename Guard,
          typename Fn,
          typename Policy>
boost::python::object with_aux(Fn fn, const Policy& policy)
{
  // Obtain the components of the Fn.  This will decompose non-member
  // and member functions into an mpl sequence.
  //   R (*)(A1)    => R, A1
  //   R (C::*)(A1) => R, C*, A1
  typedef typename mpl_signature<Fn>::type mpl_signature_type;

  // Synthesize the components into a function type.  This process
  // causes member functions to require the instance argument.
  // This is necessary because member functions will be explicitly
  // provided the 'self' argument.
  //   R, A1     => R (*)(A1)
  //   R, C*, A1 => R (*)(C*, A1)
  typedef typename boost::function_types::function_type<
      mpl_signature_type>::type signature_type;

  // Create a callable boost::python::object that delegates to the
  // guarded_function.
  return boost::python::make_function(
    guarded_function<signature_type, Guard>(fn),
    policy, mpl_signature_type());
}

} // namespace detail

/// @brief Create a callable object with guards.
template <typename Guard,
          typename Fn,
          typename Policy>
boost::python::object with(const Fn& fn, const Policy& policy)
{
  return detail::with_aux<Guard>(fn, policy);
}

/// @brief Create a callable object with guards.
template <typename Guard,
          typename Fn>
boost::python::object with(const Fn& fn)
{
  return with<Guard>(fn, boost::python::default_call_policies());
}

/// @brief Mockup class with member functions.
class spam
{
public:
  void action()
  {
    std::cout << "spam::action()"  << std::endl;
  }

  int times_two(int x)
  {
    std::cout << "spam::times_two()" << std::endl;
    return 2 * x;
  }
};

// Mockup non-member functions.
void action()
{
  std::cout << "action()"  << std::endl;
}

int times_two(int x)
{
  std::cout << "times_two()" << std::endl;
  return 2 * x;
}

/// @brief Guard that will unlock the GIL upon construction, and
///        reacquire it upon destruction.
struct no_gil
{
public:
  no_gil()  { state_ = PyEval_SaveThread(); 
              std::cout << "no_gil()" << std::endl; }
  ~no_gil() { std::cout << "~no_gil()" << std::endl;
              PyEval_RestoreThread(state_); }
private:
  PyThreadState* state_;
};

/// @brief Guard that prints to std::cout.
struct echo_guard 
{
  echo_guard()  { std::cout << "echo_guard()" << std::endl;  }
  ~echo_guard() { std::cout << "~echo_guard()" << std::endl; }
};

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;

  // Expose class and member-function.
  python::class_<spam>("Spam")
    .def("action", &spam::action)
    .def("times_two", with<no_gil>(&spam::times_two))
    ;

  // Expose non-member function.
  python::def("action", &action);
  python::def("times_two", with<boost::tuple<no_gil, echo_guard> >(
      &times_two));
}

And its usage:

>>> import example
>>> spam = example.Spam()
>>> spam.action()
spam::action()
>>> spam.times_two(5)
no_gil()
spam::times_two()
~no_gil()
10
>>> example.action()
action()
>>> example.times_two(21)
no_gil()
echo_guard()
times_two()
~echo_guard()
~no_gil()
42

Notice how multiple guards can be provided by using a container type, such as boost::tuple:

  python::def("times_two", with<boost::tuple<no_gil, echo_guard> >(
      &times_two));

When invoked in Python, example.times_two(21) produces the following output:

no_gil()
echo_guard()
times_two()
~echo_guard()
~no_gil()
42

More Related Contents:

  1. How can I implement a C++ class in Python, to be called by C++?
  2. Why is std::function not equality comparable?
  3. Using boost thread and a non-static class function
  4. boost.python not supporting parallelism?
  5. C++ Function Callbacks: Cannot convert from a member function to a function signature
  6. Compilation error: expected constructor, destructor, or type conversion before ‘;’ token [closed]
  7. Return array in a function
  8. Why doesn’t C++ support functions returning arrays?
  9. When to Overload the Comma Operator?
  10. Cross-platform way to get line number of an INI file where given option was found
  11. Why are Python Programs often slower than the Equivalent Program Written in C or C++?
  12. Which Boost libraries are header-only?
  13. Is there a way to test whether a C++ class has a default constructor (other than compiler-provided type traits)?
  14. Equivalent C++ to Python generator pattern
  15. When is a function try block useful?
  16. Boost ASIO streambuf
  17. C++ N nested vectors at runtime
  18. C++ & Boost: encode/decode UTF-8
  19. undefined behaviour somewhere in boost::spirit::qi::phrase_parse
  20. Function with same name but different signature in derived class not found
  21. Possible causes for Boost not being found by CMake in certain situations?
  22. Creating Library with backward compatible ABI that uses Boost
  23. Can Cython code be compiled to a dll so C++ application can call it?
  24. Can I treat a specific warning as an error?
  25. C++ Serialization Performance
  26. How do I take ownership of an abandoned boost::interprocess::interprocess_mutex?
  27. Boost::Spirit::QI parser: index of parsed element
  28. How to enable_shared_from_this of both parent and derived
  29. Conversion from boost::shared_ptr to std::shared_ptr?
  30. Using Boost adaptors with C++11 lambdas

Leave a Comment