+ RING-NeRF : A Versatile Architecture based on Residual Implicit Neural Grids +
+-
+
- + + Doriand Petit¹² + + +
- + + Steve Bourgeois¹ + + +
- + + Dumitru Pavel¹ + + +
- + + Vincent Gay-Bellile¹ + + +
- + + Florian Chabot¹ + + +
- + + Loïc Barthe² + + + ¹Université Paris-Saclay, CEA List, F-91120, Palaiseau, France ²CNRS, IRIT, Université de Toulouse, France +
+ Abstract +
++Since their introduction, Neural Fields have become very popular for 3D reconstruction and new view synthesis. Recent researches focused on accelerating the process, as well as improving the robustness to variation of the observation distance and limited number of supervised viewpoints. However, those approaches often led to dedicated solutions that cannot be easily combined. To tackle this issue, we introduce a new simple but efficient architecture named RING-NeRF, based on Residual Implicit Neural Grids, that provides a control on the level of detail of the mapping function between the scene and the latent spaces. Associated with a distance-aware forward mapping mechanism and a continuous coarse-to-fine reconstruction process, our versatile architecture demonstrates both fast training and state-of-the-art performances in terms of: (1) anti-aliased rendering, (2) reconstruction quality from few supervised viewpoints, and (3) robustness in the absence of appropriate scene-specific initialization for SDF-based NeRFs. We also demonstrate that our architecture can dynamically add grids to increase the details of the reconstruction, opening the way to adaptive reconstruction.
++ Global Scheme +
+
+ + Overview of RING-NeRF: to render a pixel, the casted cone is sampled with cubes. Depending on the cube volume, the corresponding LOD of the scene is selected and the latent feature is computed from the Residual Implicit Neural Grids accordingly. The density and color of the cube are first decoded from the latent feature with a tiny MLP and then integrated with other samples through volume rendering.
++ A NERF architecture which induces Level Of Details by design +
++ While a Neural Field based on a single 3D feature grid defines a single mapping function between the MLP-decoder feature space and the 3D scene space, our architecture, with its 3D hierarchical grid linked with residual connections, is able to define such a function continuously in the scale space. The architecture itself induces, by construction, a notion of level of details in the representation, allowing to reconstruct a scene at multiple precisions even if the reconstruction is only supervised at the highest level of details. +
+ + +
+ First Grid Only
+
+ All 8 Grids
++ The model illustrated here contains 8 grids of increasing resolution which have all been used jointly for the training at maximum precision. Even without specific supervision, any LOD in-between 1 and 8 can be chosen to reconstruct the scene with the corresponding resolution. +
++ Moreover, unlike other architectures that rely on a decoder latent-space which is not invariant to the position in the 3D scene space due to the use of positional encoding (eg. NGLOD) nor invariant to the level of details due to the use of feature concatenation (eg. Instant-NGP) , downweigthing related the LOD (eg. Zip-NeRF) or use of per-level MLP-decoder (eg. NGLOD, MFLOD,…), our architecture relies on a latent space that is invariant to both of them. This property makes our architecture particularly suitable for incremental reconstruction in space and level of details. +
++
+ Distance-Aware Forward Mapping for Fast Anti-Aliazing +
++ We also use the properties of our architecture to propose an adapted distance-aware forward mapping to easily solve aliasing issues. +
+ The nerfacto baseline (right) cannot easily interact with images of different resolutions and/or varying distances of observation. This results in blurred renderings when looking at high resolution renderings and the apparition of aliasing artifacts at low resolution. In comparison, RING-NeRF (left) better adapts to these parameters and produces better renderings in both high and low resolution (respectively near and far observations). + +
++ Similarly to Zip-NeRF, our solution uses cone-casting to adjust the level of details depending on the distance of the sample to the camera. However, unlike Zip-NeRF, the LOD is directly captured by our scene representation. Consequently, our solution is much faster than Zip-NeRF for both reconstruction and rendering since it does not need to achieve multi-sampling, while providing equivalent quality. + +
++ Continuous Coarse-To-Fine Optimization for more Robust NeRF models +
++ NeRF models often lack robustness when facing harder-than-usual setups, such as few viewpoints or when foregoing the scene-specific initialization in surface-based reconstruction. While we demonstrated that RING-NeRF is stabler in itself than the Instant-NGP-based nerfacto, we also proposed an adapted coarse-to-fine training process useful to bring much more robustness. While some works already used similar progressive regularization with Instant-NGP-based models (eg. NeuralAngelo), we benefit from RING-NeRF's properties to obtain a more adapted continuous process which progressively uses more degrees of freedom for the mapping function in the decoder latent space rather than learn new dimensions of it from scratch like concatenation-based architectures. +
+ We demonstrate the increased stability first on few viewpoints setups. Combined with a simple density loss taken from FreeNeRF to reduce near-cameras artifacts, this results in more 3D-coherent reconstructions when using few views than the adapted nerfacto+ baseline (nerfacto with coarse-to-fine and the same density loss). Note that the unadapted nerfacto does not succeed in creating a coherent geometry, as it mostly overfits in front of the training cameras. +
+
+ + SDF reconstruction is known to be a more unstable process than density-based NeRF reconstruction, requiring a scene-specific initialization to converge, as it adds an Eikonal constraint on the model's output. This scene-specific initialization becomes an issue in complex environments and incremental setups, where several types of scenes can coexist and are not necessarily known beforehand. In light of our previously shown increase of stability, we also demonstrated that RING-NeRF permits to forego the initialization of our model while keeping both a precision close to initialization-required models and the convergence speed of faster models (eg. Neus-Facto) which fail without initialization. +
+| + + + | +
+
+
+
+ |
+
+ Citation +
++ Acknowledgements +
+
+ This publication was made possible by the use of the CEA List FactoryIA supercomputer, financially supported by the Ile-de-France Regional Council
+
+ The website template was borrowed from Michaël Gharbi, Ref-NeRF and nerfies.
+