We propose two novel ideas (adoption of deferred rendering and mesh-based representation) to improve the quality of 3D Gaussian splatting (3DGS) based inverse rendering. We first report a problem incurred by hidden Gaussians, where Gaussians beneath the surface adversely affect the pixel color in the volume rendering adopted by the existing methods. In order to resolve the problem, we propose applying deferred rendering and report new problems incurred in a naive application of deferred rendering to the existing 3DGS-based inverse rendering. In an effort to improve the quality of 3DGS-based inverse rendering under deferred rendering, we propose a novel two-step training approach which (1) exploits mesh extraction and utilizes a hybrid mesh-3DGS representation and (2) applies novel regularization methods to better exploit the mesh. Our experiments show that, under relighting, the proposed method offers significantly better rendering quality than the existing 3DGS-based inverse rendering methods. Compared with the SOTA voxel grid-based inverse rendering method, it gives better rendering quality while offering real-time rendering.
我们提出了两个创新的想法（延迟渲染的采用和基于网格的表示）以提高基于3D高斯投影（3D Gaussian Splatting, 3DGS）的逆向渲染质量。首先，我们发现了隐藏高斯导致的问题，即表面下的高斯在现有方法采用的体渲染中对像素颜色产生了不利影响。为了解决这一问题，我们提出应用延迟渲染，并报告了在现有 3DGS 逆向渲染中简单应用延迟渲染所引发的新问题。 为了在延迟渲染下提高 3DGS 逆向渲染的质量，我们提出了一种创新的两步训练方法：（1）利用网格提取，并采用混合网格-3DGS 表示；（2）应用新的正则化方法，更好地利用网格。在重光照条件下，我们的实验表明，与现有的基于 3DGS 的逆向渲染方法相比，该方法提供了显著更好的渲染质量。与最先进的基于体素网格的逆向渲染方法相比，它在提供更高渲染质量的同时，还实现了实时渲染。