A Revisit of Shape Editing Techniques: From the Geometric to the Neural Viewpoint

Yuan, Yujie; Lai, Yu‐Kun; Wu, Tong; Gao, Lin; Liu, Ligang

doi:10.1007/s11390-021-1414-9

“…Editing a 3D model means deforming the shape of the model under some controls given by the user. There has been much work about the editing of explicit geometry representation [9,17], which we refer readers to a recent survey [80]. Traditional mesh deformation methods are based on Laplacian coordinates [35,57,58], Poisson equation [79], and dual Laplacian coordinates [2].…”

Section: Related Workmentioning

confidence: 99%

NeRF-Editing: Geometry Editing of Neural Radiance Fields

Yuan

¹

,

Sun

²

,

Lai

³

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

View full text Add to dashboard Cite

Implicit neural rendering, especially Neural Radiance Field (NeRF), has shown great potential in novel view synthesis of a scene. However, current NeRF-based methods cannot enable users to perform user-controlled shape deformation in the scene. While existing works have proposed some approaches to modify the radiance field according to the user's constraints, the modification is limited to color editing or object translation and rotation. In this paper, we propose a method that allows users to perform controllable shape deformation on the implicit representation of the scene, and synthesizes the novel view images of the edited scene without re-training the network. Specifically, we establish a correspondence between the extracted explicit mesh representation and the implicit neural representation of the target scene. Users can first utilize welldeveloped mesh-based deformation methods to deform the mesh representation of the scene. Our method then utilizes user edits from the mesh representation to bend the camera rays by introducing a tetrahedra mesh as a proxy, obtaining the rendering results of the edited scene. Extensive experiments demonstrate that our framework can achieve ideal editing results not only on synthetic data, but also on real scenes captured by users.

show abstract

“…Recent works have developed deep learning frameworks for 3D voxel grids [Choy et al 2016;Maturana and Scherer 2015], multi-view 2D rendering of 3D data [Kalogerakis et al 2017;Su et al 2018, 3D point clouds [Achlioptas et al 2018;Fan et al 2017;Qi et al 2017a,b], 3D polygonal meshes Groueix et al 2018;, and 3D implicit functions [Chen and Zhang 2019;Mescheder et al 2019;. For more detailed discussion and comparison, we refer the readers to these surveys [Ahmed et al 2018;Ioannidou et al 2017;Jin et al 2020;Xiao et al 2020;Yuan et al 2021].…”

Section: D Shape Representationsmentioning

confidence: 99%

DSG-Net: Learning Disentangled Structure and Geometry for 3D Shape Generation

Yang

¹

,

Mo

²

,

Lai

³

et al. 2022

Self Cite

View full text Add to dashboard Cite

3D shape generation is a fundamental operation in computer graphics. While significant progress has been made, especially with recent deep generative models, it remains a challenge to synthesize high-quality shapes with rich geometric details and complex structures, in a controllable manner. To tackle this, we introduce DSG-Net, a deep neural network that learns a disentangled structured & geometric mesh representation for 3D shapes, where two key aspects of shapes, geometry and structure, are encoded in a synergistic manner to ensure plausibility of the generated shapes, while also being disentangled as much as possible. This supports a range of novel shape generation applications with disentangled control, such as interpolation of structure (geometry) while keeping geometry (structure) unchanged. To achieve this, we simultaneously learn structure and geometry through variational autoencoders (VAEs) in a hierarchical manner for both, with bijective mappings at each level. In this manner, we effectively encode geometry and structure in separate latent spaces, while ensuring their compatibility: the structure is used to guide the geometry and vice versa. At the leaf level, the part geometry is represented using a conditional part VAE, to encode high-quality geometric details, guided by the structure context as the condition. Our method not only supports controllable generation applications, but also produces high-quality synthesized shapes, outperforming state-of-the-art methods.

show abstract

“…Editing a 3D model means deforming the shape of the model under some controls given by the user. There has been much work about the editing of explicit geometry representation [9,17], which we refer readers to a recent survey [80]. Traditional mesh deformation methods are based on Laplacian coordinates [35,57,58], Poisson equation [79], and dual Laplacian coordinates [2].…”

Section: Related Workmentioning

confidence: 99%

“…The mesh representation, as a kind of explicit representation, is commonly used in shape modeling and rendering. There is a lot of research work on mesh deformation or editing [80]. However, it is difficult to obtain an accurate explicit representation of a real-world scene.…”

Section: Introductionmentioning

confidence: 99%

NeRF-Editing: Geometry Editing of Neural Radiance Fields

Yuan¹,

Sun²,

Lai³

et al. 2022

Preprint

Self Cite

0

View full text Add to dashboard Cite

Implicit neural rendering, especially Neural Radiance Field (NeRF), has shown great potential in novel view synthesis of a scene. However, current NeRF-based methods cannot enable users to perform user-controlled shape deformation in the scene. While existing works have proposed some approaches to modify the radiance field according to the user's constraints, the modification is limited to color editing or object translation and rotation. In this paper, we propose a method that allows users to perform controllable shape deformation on the implicit representation of the scene, and synthesizes the novel view images of the edited scene without re-training the network. Specifically, we establish a correspondence between the extracted explicit mesh representation and the implicit neural representation of the target scene. Users can first utilize welldeveloped mesh-based deformation methods to deform the mesh representation of the scene. Our method then utilizes user edits from the mesh representation to bend the camera rays by introducing a tetrahedra mesh as a proxy, obtaining the rendering results of the edited scene. Extensive experiments demonstrate that our framework can achieve ideal editing results not only on synthetic data, but also on real scenes captured by users.

show abstract

A Revisit of Shape Editing Techniques: From the Geometric to the Neural Viewpoint

Cited by 20 publications

References 301 publications

NeRF-Editing: Geometry Editing of Neural Radiance Fields

NeRF-Editing: Geometry Editing of Neural Radiance Fields

DSG-Net: Learning Disentangled Structure and Geometry for 3D Shape Generation

NeRF-Editing: Geometry Editing of Neural Radiance Fields

Contact Info

Product

Resources

About