Screen to world: how to get XYZ point in the world from mouse cursor XY position on screen?

[fghoussen]

From the mouse cursor XY position on the screen, I need to retrieve the associated XYZ point in the world (= 3D scene). Is there an existing way/method in libigl to do this?

As I didn't find any, I tried to use glm::unProject.

First, I get camera related information from libigl viewer (view, projection, viewport - didn't find model).

        igl::opengl::glfw::Viewer viewer;

        ...

        Eigen::Vector4f igl_viewport = viewer->core().viewport;
        glm::vec4 glm_viewport(igl_viewport[0], igl_viewport[1], igl_viewport[2], igl_viewport[3]);

        Eigen::Matrix4f igl_view = viewer->core().view;
        glm::mat4 glm_view;
        for (size_t i = 0; i < 4; ++i) {
            for (size_t j = 0; j < 4; ++j) {
                glm_view[i][j] = igl_view(j, i); // Transposing matrices as Eigen mat are col-major ordered.
            }
        }

        Eigen::Matrix4f igl_proj = viewer->core().proj;
        glm::mat4 glm_proj;
        for (size_t i = 0; i < 4; ++i) {
            for (size_t j = 0; j < 4; ++j) {
                glm_proj[i][j] = igl_proj(j, i); // Transposing matrices as Eigen mat are col-major ordered.
            }
        }

Then, I get mouse cursor XY position on the screen using imgui:

        ImGuiIO& io = ImGui::GetIO();
        ImGui::Text("Mouse pos: (%g, %g)", io.MousePos.x, io.MousePos.y);
        float winX = io.MousePos.x;
        float winY = glm_viewport[3] - io.MousePos.y;
        float depth = 0.; // Point lies on the screen.
        //glReadPixels(winX, winY- cursorY, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &depth);
        glm::vec3 screen(winX, winY, depth);

Is there a way to get mouse cursor XY position from igl::opengl::glfw::Viewer or igl methods? Ideally with associated depth?

Now using glm:

        glm::vec3 world = glm::unProject(screen, glm_view, glm_proj, glm_viewport);
        ImGui::Text("XYZ: (%g, %g, %g)", world.x, world.y, world.z);

Is this the right way to do this? I get XYZ positions that doesn't sounds correct.

Any help is appreciated. I can provide more info if needed.