Subclass python class in C++

[Xfel]

I'm trying to create a C++ class that inherits from a python class. However, it does not work. Here is a short example of what i'm trying to do:

foo.py:

class Foo():
    def test():
         raise NotImplementedError

bar.cpp

#include <pybind11/pybind11.h>
#include <iostream>
namespace py = pybind11;

class FooImpl {
public:
    void test() {
        std::cout << "Hello world!" << std::endl;
    }
};

PYBIND11_MODULE(bar, m) {
    py::object foo = (py::object) py::module::import("foo").attr("Foo");
    py::class_<FooImpl>(m, "FooImpl", foo)
        .def(py::init())
        .def("test", &FooImpl::test);
}

When importing the bar module, I get multiple errors. First of all, I get a failed assertion in https://github.com/pybind/pybind11/blob/master/include/pybind11/detail/class.h#L604. This can be solved by adding the py::dynamic_attr() tag to the class definition. However, then I get a segmentation fault in https://github.com/pybind/pybind11/blob/master/include/pybind11/pybind11.h#L916.

So I wonder if this could be supported directly. Of course I could just write my c++ class, and then make a thin wrapper in python inheriting from the python class, but that is just a lot more tedious.

[nico-meetprizm]

I tried to do the same (see #1170) and failed.
If you find a way to do it, please let me know.

Good luck.

[EricCousineau-TRI]

What sort of things do you need to do with your C++-extended Python class?
Is it purely for passing it back to Python interfaces, or will you also be manipulating the C++ interface from C++ as well?

Of course I could just write my c++ class, and then make a thin wrapper in python inheriting from the python class

You could take this route, but automate either with monkey patching or codegen (with Python, I'd just go with monkey patching). You could make some function called inherit_shim(base_cls, unrelated_cls), that could return a class that inherits from base_cls, and then dynamically checks for methods, descriptors, etc., and overrides them, passing the methods back to an unrelated_cls (which could be constructed with a pointer to base_cls).

You could potentially do this all in Python, and not need to worry anything about anything C++ / pybind-specific.

For example, I'm prototyping a way to expose template parameters to Python for functions, classes, methods, etc. using this kind of mechanism:
https://github.com/EricCousineau-TRI/repro/blob/6659e31/python/bindings/pymodule/tpl/cpp_template.py
https://github.com/EricCousineau-TRI/repro/blob/6659e31/python/bindings/pymodule/tpl/cpp_template.h
https://github.com/EricCousineau-TRI/repro/blob/6659e31/python/bindings/pymodule/tpl/test/_cpp_template_test.cc
https://github.com/EricCousineau-TRI/repro/blob/6659e31/python/bindings/pymodule/tpl/test/cpp_template_test.py
https://github.com/EricCousineau-TRI/repro/blob/6659e31/python/bindings/pymodule/tpl/test/scalar_type_test.py#L40

[wjakob]

pybind11 doesn't allow this, and it's unlikely to change in the future.
However, what you can do is to create a Python class manually (and inherit from an existing class), while adding pybind11-bound methods:

py::object parent_class = py::module::import("package").attr("ParentClass");
py::object parent_metaclass = py::reinterpret_borrow<py::object>((PyObject *) &PyType_Type);
py::dict attributes;

attributes["test"] = py::cpp_function(
    [](py::object self, py::object x) {
        py::print(x);
    },
    py::arg("x"),
    py::is_method(py::none())
);

m.attr("MyClass") = parent_metaclass("MyClass", py::make_tuple(parent_class), attributes);

[YannickJadoul]

@TkTech suggested I'd add my own hack here (discussed on Gitter), for future reference:

You can, if you really want, do the following horrible thing, given some class_<...> binding and a handle typeHandle (or probably py::object would also work?) to set the __bases__ property of a class:

binding.attr("__bases__") = py::make_tuple(typeHandle) + binding.attr("__bases__");

• Warning: It's a horrible hack, not officially supported by pybind11. If you use it, do look into what it actually does and don't come complaining if your code blows up in your face.
• Another warning: I personally use it in my project to add a mixin base class (i.e., just a few methods, and no attributes), so it's not really the full kind of subclassing that this issue is about. It might work, it might not, I don't know.

That being said/warned: have fun.
And if this turns out to work decently, do let us know; who knows, maybe it could be added to pybind11 at some point.

[TkTech]

Unfortunately, @YannickJadoul approach does not work when bases have incompatible base types. For example, you cannot use this to mixin collections.abc.Sequence to your custom array type.

[YannickJadoul]

That's a pity, but not unexpected, maybe. At any rate, the other suggested solution should still work?

[josephbirkner]

@YannickJadoul your suggestion worked great for my (rather simple) use-case, thanks!

[stellaraccident]

pybind11 doesn't allow this, and it's unlikely to change in the future.
However, what you can do is to create a Python class manually (and inherit from an existing class), while adding pybind11-bound methods:

py::object parent_class = py::module::import("package").attr("ParentClass");
py::object parent_metaclass = py::reinterpret_borrow<py::object>((PyObject *) &PyType_Type);
py::dict attributes;

attributes["test"] = py::cpp_function(
    [](py::object self, py::object x) {
        py::print(x);
    },
    py::arg("x"),
    py::is_method(py::none())
);

m.attr("MyClass") = parent_metaclass("MyClass", py::make_tuple(parent_class), attributes);

I had occasion to adapt this approach recently and thought I would share: https://github.com/llvm/circt/blob/faa0e1827a116292156ff32260a195b1caf3cdce/lib/Bindings/Python/MLIRPybindAdaptors.h#L220

And where it is used: https://github.com/llvm/circt/blob/faa0e1827a116292156ff32260a195b1caf3cdce/lib/Bindings/Python/ESIModule.cpp#L113

I didn't spend a great deal of time shaving all of the edges off of it, but it feels like something that could be made relatively general and usable.

[stuhood]

The method described in #1193 (comment) does not seem to allow for passing kwargs: #5406

Is there another way to accomplish that?