Native python extensions using OCP - issues with casting [was: Getting a Reference to the Underlying OCCT C++ Instance]

[jmwright]

I've written an experimental C++ app that uses the Python interpreter and the OCCT 3D viewer together so that it can be a visual REPL for CadQuery. I have implemented a C++ callback for show_object so that the Python code can call show_object(res) to pass the cadquery.Workplane or cadquery.Assembly object to the C++ side. Things work up to a point, but I have struggled getting an OCCT TopoDS_Solid/Shape from OCP to pass to the viewer code. Is there an equivalent of the wrapped attribute on CadQuery objects that will return the underlying C++ instance? Below is a rough implementation of the show_object callback where I am just trying to use pybind11 to get a reference to the underlying OCCT instance to pass to the viewer, which gives me an error that the OCP type cannot be cast to the OCCT type.

static PyObject* show_show_object(PyObject *self, PyObject *args)
{
    // Get the argument back from the Python show_object call
    PyObject* i;
    if(!PyArg_ParseTuple(args, "O", &i))
        return NULL;

    // Handle a Workplane object differently than an Assembly object
    PyTypeObject* type = i->ob_type;
    if (strcmp(type->tp_name, "Workplane") == 0) {
        printf("cadquery.Workplane object detected.\n");
        PyObject* obj = PyObject_CallMethod(i, "val", NULL);
        if (obj == NULL) {
            printf("Error getting wrapped object from cq.Workplane.\n");
        }
        else {
            PyObject* wrapped = PyObject_GetAttrString(obj, "wrapped");
            if (wrapped != NULL) {
                if (strcmp(wrapped->ob_type->tp_name, "OCP.TopoDS.TopoDS_Solid") == 0) {
                    printf("Found OCP TopoDS_Solid object.\n");
                    py::handle h = wrapped;
                    TopoDS_Shape* x = *py::cast<TopoDS_Shape>(h);
                }
                else {
                    printf("Found some other TopoDS object.\n");
                }
            }
        }
    }
    else if (strcmp(type->tp_name, "Assembly") == 0) {
        printf("cadquery.Assembly object detected.\n");
    }

    return args;
}

The output, including the error, is below.

cadquery.Workplane object detected.
Found OCP TopoDS_Solid object.
terminate called after throwing an instance of 'pybind11::cast_error'
  what():  Unable to cast Python instance of type <class 'OCP.TopoDS.TopoDS_Solid'> to C++ type 'TopoDS_Shape'
Aborted

[adam-urbanczyk]

I never used it, but take a look here:
https://pybind11.readthedocs.io/en/stable/advanced/pycpp/object.html#casting-back-and-forth
It seems that something like this should work:

TopoDS_Solid *x = py::cast<TopoDS_Solid*>(h);

You seem to be mixing raw python api and pybind11, there might be some issues with it btw.

[jmwright]

Thanks @adam-urbanczyk . I get an error that there is invalid use of incomplete type ‘class TopoDS_Solid’ during the cast. If I try to use the superclass TopoDS_Shape, the program will compile, but I get the runtime error I reference above about not being able to cast OCP.TopoDS.TopoDS_Solid to TopoDS_Solid. I'll spend some time with the link you provided and see if I can make any progress. I'm putting the trace from the build error below for completeness.

In file included from /usr/include/c++/11/bits/move.h:57,
                 from /usr/include/c++/11/bits/stl_pair.h:59,
                 from /usr/include/c++/11/bits/stl_algobase.h:64,
                 from /usr/include/c++/11/bits/specfun.h:45,
                 from /usr/include/c++/11/cmath:1935,
                 from /usr/include/c++/11/math.h:36,
                 from /usr/include/python3.10/pyport.h:210,
                 from /usr/include/python3.10/Python.h:50,
                 from /home/jwright/Downloads/func_inject/main.cpp:2:
/usr/include/c++/11/type_traits: In instantiation of ‘struct std::is_base_of<pybind11::detail::pyobject_tag, TopoDS_Solid>’:
/usr/include/pybind11/cast.h:828:68:   recursively required by substitution of ‘template<class T> class pybind11::detail::type_caster<T, typename std::enable_if<std::is_base_of<pybind11::detail::pyobject_tag, typename std::remove_reference<_Tp>::type>::value, void>::type> [with T = TopoDS_Solid]’
/usr/include/pybind11/cast.h:828:68:   required by substitution of ‘template<class T> struct pybind11::detail::move_always<T, typename std::enable_if<std::integral_constant<bool, (pybind11::detail::satisfies_none_of<T, std::is_void, std::is_pointer, std::is_reference, std::is_const>::value && (pybind11::detail::negation<pybind11::detail::is_copy_constructible<T1> >::value && (std::is_move_constructible<_Tp>::value && std::is_same<decltype (declval<pybind11::detail::type_caster<typename pybind11::detail::intrinsic_type<T>::type, void> >().operator T&()), T&>::value)))>::value, void>::type> [with T = TopoDS_Solid*]’
/usr/include/pybind11/detail/common.h:681:30:   required by substitution of ‘template<class T> std::enable_if_t<pybind11::detail::negation<std::integral_constant<bool, (pybind11::detail::move_always<T>::value || pybind11::detail::move_if_unreferenced<T>::value)> >::value, T> pybind11::cast(pybind11::object&&) [with T = TopoDS_Solid*]’
/home/jwright/Downloads/func_inject/main.cpp:79:62:   required from here
/usr/include/c++/11/type_traits:1422:38: error: invalid use of incomplete type ‘class TopoDS_Solid’
 1422 |     : public integral_constant<bool, __is_base_of(_Base, _Derived)>
      |                                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/jwright/Downloads/func_inject/main.cpp:6:
/usr/local/include/opencascade/TopoDS.hxx:29:7: note: forward declaration of ‘class TopoDS_Solid’
   29 | class TopoDS_Solid;
      |       ^~~~~~~~~~~~
In file included from /usr/include/pybind11/cast.h:16,
                 from /usr/include/pybind11/attr.h:13,
                 from /usr/include/pybind11/pybind11.h:13,
                 from /home/jwright/Downloads/func_inject/main.cpp:8:
/usr/include/pybind11/detail/type_caster_base.h: In instantiation of ‘pybind11::detail::type_caster_base<type>::type_caster_base() [with type = TopoDS_Solid]’:
/usr/include/pybind11/cast.h:33:56:   required from ‘pybind11::detail::make_caster<T> pybind11::detail::load_type(const pybind11::handle&) [with T = TopoDS_Solid*; pybind11::detail::make_caster<T> = pybind11::detail::type_caster<TopoDS_Solid, void>]’
/usr/include/pybind11/cast.h:892:35:   required from ‘T pybind11::cast(const pybind11::handle&) [with T = TopoDS_Solid*; typename std::enable_if<(! std::is_base_of<pybind11::detail::pyobject_tag, typename std::remove_reference<_Tp>::type>::value), int>::type <anonymous> = 0]’
/home/jwright/Downloads/func_inject/main.cpp:79:62:   required from here
/usr/include/pybind11/detail/type_caster_base.h:902:43: error: invalid use of incomplete type ‘class TopoDS_Solid’
  902 |     type_caster_base() : type_caster_base(typeid(type)) { }
      |                                           ^~~~~~~~~~~~
In file included from /home/jwright/Downloads/func_inject/main.cpp:6:
/usr/local/include/opencascade/TopoDS.hxx:29:7: note: forward declaration of ‘class TopoDS_Solid’
   29 | class TopoDS_Solid;
      |       ^~~~~~~~~~~~
In file included from /usr/include/pybind11/detail/type_caster_base.h:16,
                 from /usr/include/pybind11/cast.h:16,
                 from /usr/include/pybind11/attr.h:13,
                 from /usr/include/pybind11/pybind11.h:13,
                 from /home/jwright/Downloads/func_inject/main.cpp:8:
/usr/include/pybind11/detail/typeid.h: In instantiation of ‘std::string pybind11::type_id() [with T = TopoDS_Solid; std::string = std::__cxx11::basic_string<char>]’:
/usr/include/pybind11/cast.h:872:87:   required from ‘pybind11::detail::type_caster<T, SFINAE>& pybind11::detail::load_type(pybind11::detail::type_caster<T, SFINAE>&, const pybind11::handle&) [with T = TopoDS_Solid; SFINAE = void]’
/usr/include/pybind11/cast.h:880:14:   required from ‘pybind11::detail::make_caster<T> pybind11::detail::load_type(const pybind11::handle&) [with T = TopoDS_Solid*; pybind11::detail::make_caster<T> = pybind11::detail::type_caster<TopoDS_Solid, void>]’
/usr/include/pybind11/cast.h:892:35:   required from ‘T pybind11::cast(const pybind11::handle&) [with T = TopoDS_Solid*; typename std::enable_if<(! std::is_base_of<pybind11::detail::pyobject_tag, typename std::remove_reference<_Tp>::type>::value), int>::type <anonymous> = 0]’
/home/jwright/Downloads/func_inject/main.cpp:79:62:   required from here
/usr/include/pybind11/detail/typeid.h:50:22: error: invalid use of incomplete type ‘class TopoDS_Solid’
   50 |     std::string name(typeid(T).name());
      |                      ^~~~~~~~~
In file included from /home/jwright/Downloads/func_inject/main.cpp:6:
/usr/local/include/opencascade/TopoDS.hxx:29:7: note: forward declaration of ‘class TopoDS_Solid’
   29 | class TopoDS_Solid;
      |       ^~~~~~~~~~~~
make[2]: *** [CMakeFiles/func-inj.dir/build.make:76: CMakeFiles/func-inj.dir/main.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/func-inj.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

[adam-urbanczyk]

Incomplete type usually means that you did not include the relevant header #include <TopoDS_Solid.hxx>

[jmwright]

Even with that include I still get a similar runtime error.

With this line:

TopoDS_Solid *x = py::cast<TopoDS_Solid*>(h);

I get this runtime error:

terminate called after throwing an instance of 'pybind11::cast_error'
  what():  Unable to cast Python instance of type <class 'OCP.TopoDS.TopoDS_Solid'> to C++ type 'TopoDS_Solid'
Aborted

[adam-urbanczyk]

I tested a simple example with using pybind11 only APIs and casting seems to work as expected. I don't know what is wrong with your snippet, but I'd personally begin with using the pybind11 api only (args and embedding).

[whophil]

@jmwright are you compiling OCP and your own project together, or using pre-compiled OCP from somewhere else?

[jmwright]

@whophil Somewhere else. I have the interpreter that is embedded in my project pointing at a Python virtual environment where OCP is installed. I want that flexibility, although it requires me to make sure the Python version in the virtual environment is the same as the compiled-in Python version.

Thanks @adam-urbanczyk . I had read that PyObject and py::handle are basically the same thing, but I could see how mixing stock Python embedding with pybind11 could cause weird issues. I'll move everything to pybind and try that.

[whophil]

@jmwright not sure if what you're seeing is the same issue, but in a private project in which I use OCP classes in C++ code, I would run into the same issue until I used:

• the exact same compiler and version
• the exact same pybind version
  For OCP and my project.

In my case where I am installing OCP from conda-forge, that means finding the exact compiler and pybind version used for the specific OCP build being installed.

[jmwright]

Thanks for the tip @whophil . I may try building OCP then if the switch to pure pybind11 embedding yields the same problem. That way I will know that everything is on the same versions.

[jmwright]

Here is my MRE with pure pybind11 that results in the same error:

#include <TopoDS_Solid.hxx>

#include <pybind11/embed.h>

namespace py = pybind11;

PYBIND11_EMBEDDED_MODULE(show, m) {
    // Also tried `m.def("show_object", [](TopoDS_Solid s) {`
    m.def("show_object", [](py::object s) {
        TopoDS_Solid x = py::cast<TopoDS_Solid>(s);
    });
}

int main(int argc, char *argv[])
{
    // Start the Python interpreter
    py::scoped_interpreter guard{};

    // Module that allows us to provide show_object
    auto py_module = py::module_::import("show");

    py::exec(R"(
        import show
        show_object = show.show_object
        import cadquery as cq
        res = cq.Workplane().box(10, 10, 10)
        show_object(res.val().wrapped)
    )");
}

Here is the full error message:

terminate called after throwing an instance of 'pybind11::error_already_set'
  what():  RuntimeError: Unable to cast Python instance of type <class 'OCP.TopoDS.TopoDS_Solid'> to C++ type 'TopoDS_Solid'

At:
  <string>(7): <module>

Aborted

[whophil]

@jmwright in case you are using OCP from conda-forge, here is one combination of OCP packages and compilers which works for my case - all dependencies installed from the conda-forge channel:

• Python 3.11
• OCP 7.7.0.0
• GCC 12.3.0
• pybind 2.11.1

[jmwright]

@whophil How do you set up the environment? I installed cmake, make, gcc and pybind11 with conda, but then make cannot find OpenGL, which I only seem to be able to install at the system level.

[jmwright]

@whophil Nevermind, I just disabled the OpenCASCADE libraries that were trying to pull in OpenGL.

I still get the same error with my build environment set up the way you are suggesting. Would you be able to confirm that you can get my minimal example above to compile and work in your environment?

[whophil]

@jmwrightYour example program does run in my build environment! But I made a mistake in the dependencies above - it should be GCC 11, not 12, for the Python 3.11 build of OCP 7.7.0.0.

Here is the environment.yml file I used for my build environment:

channels:
  - conda-forge
dependencies:
  - python 3.11
  - gxx_linux-64 11*
  - pybind11 2.11 
  - ocp 7.7.0.0
  - cmake
  - cadquery

And here is my CMakeLists.txt, if you are using CMake

cmake_minimum_required(VERSION 3.15)
project(example)

set(CMAKE_CXX_STANDARD 14)

find_package( pybind11 REQUIRED )

find_package(OpenCASCADE CONFIG REQUIRED)
link_directories(${OpenCASCADE_LIBRARY_DIR})
include_directories(${OpenCASCADE_INCLUDE_DIR})

find_package (Python3 COMPONENTS
    Interpreter
    Development.Module)

add_executable(example main.cpp)
target_link_libraries(example PRIVATE
    ${OpenCASCADE_ModelingData_LIBRARIES}
    pybind11::embed)

[jmwright]

Thanks for posting that @whophil

I get an error when trying to install based on that environment file.

$ conda env create -f environment.yml -n cq-repl
Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound:
  - gxx_linux-64==11

[whophil]

@jmwright made one more change, should be

  - gxx_linux-64 11*

[jmwright]

@whophil Thanks, that worked, and the cast now works. I guess the best move here is to compile OCP on the local system, and then statically link it in the project. That way I can use the system's OpenGL library and not have any version conflicts between my project and conda.

[whophil]

@jmwright glad it works for you.

I haven't looked into it deeply, but I think the GL/conda issue should be resolvable. This likely only matters if you intend to distribute your project through conda, though.

[adam-urbanczyk]

@whophil Thanks, that worked, and the cast now works. I guess the best move here is to compile OCP on the local system, and then statically link it in the project. That way I can use the system's OpenGL library and not have any version conflicts between my project and conda.

That sounds like an overkill. Likely you just need to use the correct version of pybind11 and the default platform compiler.

[jmwright]

@adam-urbanczyk If I change the environment.yml file to the following (allowing any gxx compiler version), the app breaks and starts to have the same cast error again.

channels:
  - conda-forge
dependencies:
  - python 3.11
  - gxx_linux-64
  - pybind11 2.11 
  - ocp 7.7.0.0
  - cmake
  - cadquery

$ conda list | grep gxx
gxx_impl_linux-64         13.2.0               h338b0a0_3    conda-forge
gxx_linux-64              13.2.0               hc53e3bf_2    conda-forge

[adam-urbanczyk]

Thanks for checking! So the final conclusion is: use the same version of pybind11 and gcc .

I'm updating the issue title to make it better findable. Some additional reading:
https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html

[whophil]

@adam-urbanczyk I think technically the pybind version doesn't need to be pinned, as long as the pybind11 ABI version is the same between OCP and the extension library. In the conda-forge world, this is ensured using the pybind11-abi metapackage. I did run into an issue in the past which I thought was related to pybind11 ABI compatibility, but was unrelated in the end. Nevertheless, the discussion in the linked thread may be of interest.

[adam-urbanczyk]

Thanks for the clarification. you are right @whophil .

@jmwright if it is not something commercially sensitive, could you maybe add a working example to the wiki?

[jmwright]

@adam-urbanczyk Which wiki?

[adam-urbanczyk]

Maybe here

[jmwright]

@adam-urbanczyk Done: https://github.com/CadQuery/OCP/wiki/Building-Native-Extensions

[jdegenstein]

Wanted to say thank you everyone for documenting this discussion here and on the wiki. I was facing the same issue on Windows builds with incompatible types. What fixed it for me was downgrading from pybind11=2.12 (released only a few weeks ago) to pybind11=2.11 which I assume is what was used to compile ocp==7.7.2 from conda-forge.