Neural Radiance Fields (NeRFs) have demonstrated the remarkable potential of neural networks to capture the intricacies of 3D objects. By encoding the shape and color information within neural network weights, NeRFs excel at producing strikingly sharp novel views of 3D objects. Recently, numerous generalizations of NeRFs utilizing generative models have emerged, expanding its versatility. In contrast, Gaussian Splatting (GS) offers a similar renders quality with faster training and inference as it does not need neural networks to work. We encode information about the 3D objects in the set of Gaussian distributions that can be rendered in 3D similarly to classical meshes. Unfortunately, GS are difficult to condition since they usually require circa hundred thousand Gaussian components. To mitigate the caveats of both models, we propose a hybrid model that uses GS representation of the 3D object's shape and NeRF-based encoding of color and opacity. Our model uses Gaussian distributions with trainable positions (i.e. means of Gaussian), shape (i.e. covariance of Gaussian), color and opacity, and neural network, which takes parameters of Gaussian and viewing direction to produce changes in color and opacity. Consequently, our model better describes shadows, light reflections, and transparency of 3D objects.
神经辐射场(NeRFs)已经展示了神经网络捕捉三维物体复杂性的惊人潜力。通过在神经网络权重中编码形状和颜色信息，NeRFs在生成三维物体的新视角方面表现出色，能产生极为清晰的图像。最近，利用生成模型的NeRFs的众多泛化版本已经出现，扩展了其多功能性。相比之下，高斯飞溅(Gaussian Splatting, GS)提供了类似的渲染质量，且由于不需要神经网络就可以工作，因此在训练和推理上更快。我们将关于三维物体的信息编码在一组高斯分布中，这些分布可以像传统网格一样在三维中渲染。不幸的是，GS很难进行条件设定，因为它们通常需要大约十万个高斯组件。为了缓解这两种模型的局限，我们提出了一种混合模型，它使用GS表示三维物体的形状，以及基于NeRF的颜色和不透明度编码。我们的模型使用具有可训练位置（即高斯的均值）、形状（即高斯的协方差）、颜色和不透明度的高斯分布，以及一个神经网络，该网络采用高斯的参数和观看方向来产生颜色和不透明度的变化。因此，我们的模型更好地描述了三维物体的阴影、光反射和透明度。