We propose GauFace, a novel Gaussian Splatting representation, tailored for efficient animation and rendering of physically-based facial assets. Leveraging strong geometric priors and constrained optimization, GauFace ensures a neat and structured Gaussian representation, delivering high fidelity and real-time facial interaction of 30fps@1440p on a Snapdragon 8 Gen 2 mobile platform. Then, we introduce TransGS, a diffusion transformer that instantly translates physically-based facial assets into the corresponding GauFace representations. Specifically, we adopt a patch-based pipeline to handle the vast number of Gaussians effectively. We also introduce a novel pixel-aligned sampling scheme with UV positional encoding to ensure the throughput and rendering quality of GauFace assets generated by our TransGS. Once trained, TransGS can instantly translate facial assets with lighting conditions to GauFace representation, With the rich conditioning modalities, it also enables editing and animation capabilities reminiscent of traditional CG pipelines. We conduct extensive evaluations and user studies, compared to traditional offline and online renderers, as well as recent neural rendering methods, which demonstrate the superior performance of our approach for facial asset rendering. We also showcase diverse immersive applications of facial assets using our TransGS approach and GauFace representation, across various platforms like PCs, phones and even VR headsets.
我们提出了GauFace，一种全新的高斯散点表示法，专为高效动画和物理基础的面部资产渲染设计。通过利用强几何先验和约束优化，GauFace确保了干净且结构化的高斯表示，能够在Snapdragon 8 Gen 2移动平台上实现1440p@30fps的高保真实时面部交互。 接着，我们介绍了TransGS，一种扩散式Transformer，用于将物理基础的面部资产快速转化为相应的GauFace表示。具体来说，我们采用了基于patch的管线来有效处理大量的高斯点。此外，我们引入了一种新颖的像素对齐采样方案，并结合UV位置编码，确保由TransGS生成的GauFace资产的吞吐量和渲染质量。一旦训练完成，TransGS能够即时将带有光照条件的面部资产转化为GauFace表示。凭借丰富的条件控制模式，它还能够实现类似传统CG管线的编辑和动画功能。 我们进行了广泛的评估和用户研究，与传统的离线和在线渲染器以及最近的神经渲染方法相比，我们的方法在面部资产渲染上表现出显著的优越性。此外，我们展示了使用TransGS方法和GauFace表示的面部资产在多个平台（如PC、手机甚至VR头戴设备）上的多样化沉浸式应用。