GLTFExporter: Provide a way to export a mesh with multiple primitives

[gkjohnson]

Description

Currently when exporting a model with GLTFLoader all geometry-material pairs are separated into different "mesh" nodes that all have a single primitives in the final GLTF file. This sacrifices a lot of control over the final file, though, since GLTF can support multiple primitives (with unique material per primitive) per mesh. When just working in three.js this may be okay but other tools interpret these multi-primitive meshes differently. Unreal, for example, will load them as a single object with multiple materials assigned which is significantly easier for an artist to work with.

My current situation is that I'm trying to simplify a complex model and merge meshes components into semantic components with multiple materials for use in Unreal using three.js but the exporter doesn't allow for control over primitives. The reason I'm using three.js is because merging geometry and reasoning about & adjusting the hierarchy is simpler than other tools I've used.

cc @donmccurdy

Solution

Provide GLTFPrimitive, GLTFMesh Subclasses

When importing a GLTF two GLTFPrimitive and GLTFMesh subclasses could be provided to better represent the original structure of the imported GLTF. Then the GLTFExporter could use these classes to export the structure reflected by these classes. This would allow for more user control of the structure, as well, since they could be created as-needed. Alternatively a flag in userData could be added to support the same thing.

Alternatives

I've looked into gltf-transform for this but managing the hierarchy is not as clear or easy to visualize.

Additional context

No response

[donmccurdy]

I'm hesitant to stray farther from the principle of returning "normal three.js objects" from GLTFLoader. While we can always do more by dropping that rule, the complexity does stack up. Similarly, we'd need to think about GLTFPoints, GLTFLines, GLTFSkinnedMesh, etc.

I already feel "not great" about how we're using .userData to store mimeType and extensions. More of that would be an option, but note that there's (currently) no guarantee that a glTF 'mesh' is represented as an object in the output at all - it might be skipped if the mesh has only one primitive.

Would the proposal at #29768 (comment) help, do you think? Part of the problem perhaps is that we're just not consistent. Knowing that GLTFLoader will reliably output a glTF Mesh as a THREE.Group, with drawable primitives as direct children, might be enough?

---

Personally I would use glTF Transform for this, but I won't take up space here on the topic unless that's of interest.

[gkjohnson]

Would the proposal at #29768 (comment) help, do you think? Part of the problem perhaps is that we're just not consistent.

This doesn't help my use case, unfortunately, since the exporter will still export each imported primitive as a separate mesh nodes again and Unreal will not use nodes with multiple materials.

Personally I would use glTF Transform for this, but I won't take up space here on the topic unless that's of interest.

I've looked into this and have used glTF Transform in other cases but it still feels fairly limited in terms of geometry manipulation. Or maybe it's just my lack of experience with the tool. For my use case I'm needing to merge both nodes and primitives with common material definitions which is fairly easy to do in three.js.

If this is definitely not going to be added I can look into using gltf transform for this use case more this week but this feels really limiting in terms of three.js' ability to export GLTF. It feels like there's distinct gap in terms of tools that give you a lot of freedom, flexibility, and ecosystem over a 3d scene (three.js) and tools that give you detailed control over the GLTF structure on export (gltf-transform). I feel like I can create a model I want in three.js but I can't export it in a way that's useful for me.

[donmccurdy]

This doesn't help my use case, unfortunately, since the exporter will still export each imported primitive as a separate mesh nodes again...

I'm OK with the idea that GLTFExporter should look for [some criteria] and export a glTF mesh containing multiple primitives when it sees that criteria. I'm just worried about the criteria being custom classes specifically, which GLTFLoader then needs to create, and which have to be supported throughout three.js. If we want to say, when GLTFExporter sees a THREE.Group containing drawable objects as direct children, it should export those as part of a single glTF Mesh, I'd be comfortable with that. We'd then probably want to update GLTFLoader as described in #29768 (comment).

For my use case I'm needing to merge both nodes and primitives with common material definitions...

A use case like "merge all materials and meshes that are compatible with one another in the scene" is something that I've implemented before in glTF Transform; operations like applying a matrix to geometry, merging geometry, re-arranging the scene graph, and doing equality comparisons on materials are built-in. But no pressure, I don't know all your requirements, if that's something we want to discuss more maybe https://github.com/donmccurdy/glTF-Transform/discussions is the place.

[gkjohnson]

I'm just worried about the criteria being custom classes specifically, which GLTFLoader then needs to create, and which have to be supported throughout three.js.

In what way do they have to be supported through three.js? The only use case I can think of is JSON exporter but if someone exports a scene to JSON and reimports it I'd think it would be okay if the glTF structure were lost. Everything else should "just work", though. "Replacing" an existing mesh with a "GLTFMeshPrimitive" class if you want to change how it's exported is more cumbersome than necessary, though, I agree.

If we want to say, when GLTFExporter sees a THREE.Group containing drawable objects as direct children, it should export those as part of a single glTF Mesh, I'd be comfortable with that.

This would work but it strikes me as unintuitive to add an unmarked empty group in order to create a "mesh" node rather than a "primitive". It's workable if you know what you're doing, though, and it would at least alleviate the issue of GLTFExporter exporting a model structure that's significantly different from what GLTFLoader imported.

Marking something as a "primitive" in a meshes userData feels more controllable to me but maybe this causes other problems. Really having some way that allows for controlling the primitive / mesh structure and retain the same type of GLTF hierarchy as was imported is my priority.

A use case like "merge all materials and meshes that are compatible with one another in the scene" is something that I've implemented before in glTF Transform;

I have no doubt these things can be done in GLTF Transform and I'm using for some other use cases (and it's great) - but I think this a matter of convenience and existing knowledge. I've used three.js for almost 10 years and it's 99% to doing what I need in thise case. My iteration speed, ability to visualize & debug my model changes, etc are all going to be a lot simpler for me in three.js than learning the ins and outs of a new tool. In other cases I can't at all do what I want in GLTF Transform without a lot of hoops - including using rendering to cull pieces of a model, raycast, etc.

[donmccurdy]

In what way do they have to be supported through three.js?

We are talking about having GLTFLoader return custom subclasses instead of core three.js class instances, correct? In forums and stack overflow we're going to get questions like "How to raycast a GLTFMesh" or "difference between Points and GLTFPointsPrimitive" or "how to clone GLTFSkinnedMeshPrimitive". Presumably gltfjsx and @threlte/gltf will need to be updated for the new types. We'll have to explain that a GLTFMesh is not actually a Mesh, it is a Group.

That seems like a big increase in conceptual overhead to me, and is something that other three.js loaders don't do. The problems caused by the fix seem worse than the problems it solves, to me.

[gkjohnson]

That seems like a big increase in conceptual overhead to me, and is something that other three.js loaders don't do.

Yeah that's fair - it's more cumbersome to restructure an existing hierarchy this way, as well. UserData fields would be simpler.

The group-hierarchical method is fine with me for now, as well.

[donmccurdy]

Ok! I think changing GLTFLoader to output a consistent group structure (#29768 (comment)) would be a prerequisite to any of these other options, we wouldn't have anywhere to put .userData fields without that. I don't feel too strongly about whether GLTFExporter should create mesh/primitive relationships based on Group usage alone, or would additionally require .userData.

[gkjohnson]

I think changing GLTFLoader to output a consistent group structure (#29768 (comment)) would be a prerequisite to any of these other options

Yeah I think this is a good change either way.

I don't feel too strongly about whether GLTFExporter should create mesh/primitive relationships based on Group usage alone, or would additionally require .userData.

After thinking about it a bit more it feels okay to model a three.js hierarchy as "any Mesh is a GLTF Primitive", which in turn means "any object with an immediate Mesh child is a GLTF Mesh". And anything else is a group parent node. So no user data fields would be needed.

Here are some examples just to make sure it all works conceptually in odd cases:

Nested Meshes

// three.js
Mesh (geometry 1)
└ Mesh (geometry 2)

// glTF
mesh node
└ primitives (geometry 1)
└ children
    └ mesh node
        └ primitives (geometry 2)

// reimported three.js
Group
└ Mesh (geometry 1)
└ Group
    └ Mesh (geometry 2)

Inconsistent Child Types

// three.js
Group
└ Mesh (geometry 1)
└ Group
    └  Mesh (geometry 2)

// glTF
mesh node
└ primitives (geometry 1)
└ children
    └ mesh node
        └ primitives (geometry 2)

// reimported three.js
Group
└ Mesh (geometry 1)
└ Group
    └ Mesh (geometry 2)

It does mean any "mesh" node names will be lost, though. But I think that's okay as long as we document how GLTFExporter interprets the three.js hierarchy unless there are some other mesh name dependencies I'm unfamiliar with.

[gkjohnson]

@donmccurdy Any advice on where to start with this? Or how significant of a change this will be? I'm less familiar with the GLTFExporter / Loader.

[donmccurdy]

I think the changes to GLTFLoader.js could be technically minimal, mainly removing the middle case here...

three.js/examples/jsm/loaders/GLTFLoader.js

Lines 4252 to 4264 in 5fa82c7

	} else if ( objects.length > 1 ) {
	node = new Group();
	} else if ( objects.length === 1 ) {
	node = objects[ 0 ];
	} else {
	node = new Object3D();
	}

... and then testing carefully that we're correctly mapping...

• .name -> .name
• .extras -> .userData
• .extensions -> .userData.gltfExtensions

... according to the node/mesh/primitive relationships discussed above. I can help create test assets for multi-primitive meshes with names, extras, extensions, etc.

/cc @drcmda this change (tl;dr in #29768 (comment)) would require an update in gltfjsx. I think it's valuable since the structure of the scene graph coming out of GLTFLoader will be much more predictable, but FYI in case this raises bigger concerns!

[donmccurdy]

I haven't looked much into the changes required in GLTFExporter, but I suspect it should be fine and would be open to any refactoring that makes sense there.