[DRAFT] Gaussian Opacity Fields implementation

[niujinshuchong]

Implementation of Gaussian Opacity Fields. Training and testing works.

Benchmark results on Mip-NeRF 360 dataset.

	PSNR	SSIM	LPIPS	num_GS (M)
classic	28.989	0.871	0.136	3.23
GOF	28.843	0.867	0.140	3.08
GOF + depth-normal consitency	28.616	0.865	0.142	3.08

Visual comparison:
without depth-normal consistency

and with depth-normal consistency

Left: rendered normal. right: normal from depth.

Extracted Mesh:

TODO:

• Mesh extraction
• Clean up
• Documation

[GuoFengYung]

Hi,

I am trying to extract mesh in the gsplat module and have written some functions for this purpose, but I am encountering some bugs. Could you please guide me on how to better implement this feature? Any suggestions or example code would be greatly appreciated.

Thank you!

Bugs:
TSDF open3d function:

[niujinshuchong]

@GuoFengYung You can refer to https://github.com/autonomousvision/gaussian-opacity-fields/blob/main/extract_mesh_tsdf.py for mesh extraction with TSDF. However, TSDF is not good at extracting fine detailed geometry or extracting the mesh at background regions. I just implemented GOF's tetrahedra meshing in this PR. You can try the latest code.

[GuoFengYung]

@GuoFengYung You can refer to https://github.com/autonomousvision/gaussian-opacity-fields/blob/main/extract_mesh_tsdf.py for mesh extraction with TSDF. However, TSDF is not good at extracting fine detailed geometry or extracting the mesh at background regions. I just implemented GOF's tetrahedra meshing in this PR. You can try the latest code.

Thanks for your reply!!
I will try it

[KohsukeIde]

Appreciate your amazing work!

I'm curious about how to proceed with from tetranerf.utils.extension import cpp. Would I need to clone and install the Tetranerf myself, or is there like a more convenient approach?

[niujinshuchong]

@KohsukeIde You can install it from https://github.com/autonomousvision/gaussian-opacity-fields/tree/main/submodules/tetra-triangulation