Cage functionality and visualization functionality request

[MrJRules]

First off, thank you for creating and maintaining such an incredible tool! Textools has been invaluable to my workflow, and I truly appreciate all the work that’s gone into it.

I’d like to suggest a feature that could save significant time, especially when baking complex meshes: Cage Extrusion and Ray Distance Visualization.

Why It’s Important

Currently, determining appropriate cage extrusion and ray distances often involves trial and error, leading to lost time and potential frustration.

A visualization tool would:

1. Save time by eliminating guesswork and debugging during baking.
2. Enhance learning by providing a clear visual aid for newcomers, making an abstract process more tangible.
3. Improve decision-making on whether a custom-built cage is necessary for specific assets.

Proposed Solution

Here’s a possible implementation for the feature:

Temporary Cage Visualization:

1. Create a temporary copy of the low-poly mesh, color it, and extrude it along the normals.
2. Ray Distance Indicators: Display raycasts as arrows extending from the low-poly mesh, stopping at the distance specified in the ray distance field.
3. Error Color Coding: Colorize cage components (edges, vertices, and faces) to indicate issues:
   • Yellow: Mesh distortions or warning areas.
   • Orange: Overlapping geometry where the cage intersects itself.
   • Red: High-poly geometry still poking through the cage.
4. Gradient Visualization:
   • Apply a gradient color across the entire cage, transitioning from green (no issues) to red (major issues).
5. Automatic Adjustment:
   • Analyze the cage and automatically expand it until the high-poly mesh no longer intersects.
   • Provide an error message if a reasonable cage cannot be generated, suggesting alternatives like exploding the geometry or building a custom cage.

Benefits

• This feature would make the baking process smoother and more intuitive while reducing time spent on manual fixes. Additionally, it would make Textools even more approachable for beginners, helping them grasp critical concepts with visual feedback.

Visualized Ideas

Below are examples of how this could look in action:

1. Mesh Distortion Visualization: Highlighting stretched areas in yellow.
2. Mesh Overlap Indicators: Showing tangled or overlapping cage geometry in orange.
3. Error Color Coding: Highlighting problematic areas in red for easy debugging.
   
   
   

Thank you for considering this suggestion! I believe it would be a fantastic addition to Textools and help both professionals and beginners alike. Please let me know if I can provide any additional information or examples to support this idea.

[franMarz]

Hello there. Thank you for this so well laid-out feature request.

I agree, Cage Extrusion and Ray Distance are hard to use in a predictable way, especially when using both together on a bake. I don't think visualization can help to better undertand how they interact, but would be great for individual usage of any of them, so it's worth trying to implement this, at least partially.

The visualization modes would be trickier to implement, though, as it would require us to use the GPU module of the API, coupled with a somehow complicated geometry analysis involving the original and newly created cage meshes, which wouldn't be easy to make to run fast on highpoly meshes. Maybe someone from the community would be open to explore it once the cage display is in place.

I want to remember everybody that you can always rely on using an actual, custom Cage object during a high to low bake for better control, and that Cage Extrusion and Ray Distance will always be weaker alternatives to that workflow, as they will always be automated core Blender routines that we can't improve nor control with an addon.