DispVoxNets: Non-Rigid Point Set Alignment with Supervised Learning Proxies

Shimada, Shigetaka; Golyanik, Vladislav; Tretschk, Edgar; Stricker, Didier; Theobalt, Christian

doi:10.1109/3dv.2019.00013

Cited by 23 publications

(24 citation statements)

References 62 publications

(122 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hence, the ideal architecture is a network which estimates the hand shape without losing local spatial information while preserving the topology of the hand shape. To achieve this, we register the estimated shape by V2S-Net to the probabilistic shape representation estimated by FCN (V2V-ShapeNet) using DispVoxNets pipeline [25].…”

Section: D Hand Shape Estimationmentioning

confidence: 99%

“…where Q and d are the voxel grid size and the ground truth displacement, respectively. Since it is difficult to obtain d between the voxelized shape VS and hand surface VT , the displacements are first computed between V T and VT , and are discretized to obtain d. For more details of ground truth voxelized grid computation, please refer to [25].…”

Section: D Hand Shape Estimationmentioning

confidence: 99%

“…In our voxel-based 3D hand shape and pose estimation pipeline (Fig. 2), the Dis-pVoxNet [25] component requires shape annotations in its source-to-target displacement field learning phase. These annotations are available only for the synthetic dataset which leaves a domain gap on the performance of Dis-pVoxNet when tested on real dataset.…”

Section: D Hand Shape Estimationmentioning

confidence: 99%

See 2 more Smart Citations

HandVoxNet: Deep Voxel-Based Network for 3D Hand Shape and Pose Estimation From a Single Depth Map

Malik

Abdelaziz

Elhayek

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

View full text Add to dashboard Cite

3D hand shape and pose estimation from a single depth map is a new and challenging computer vision problem with many applications. The state-of-the-art methods directly regress 3D hand meshes from 2D depth images via 2D convolutional neural networks, which leads to artefacts in the estimations due to perspective distortions in the images.In contrast, we propose a novel architecture with 3D convolutions trained in a weakly-supervised manner. The input to our method is a 3D voxelized depth map, and we rely on two hand shape representations. The first one is the 3D voxelized grid of the shape which is accurate but does not preserve the mesh topology and the number of mesh vertices. The second representation is the 3D hand surface which is less accurate but does not suffer from the limitations of the first representation. We combine the advantages of these two representations by registering the hand surface to the voxelized hand shape. In the extensive experiments, the proposed approach improves over the state of the art by 47.8% on the SynHand5M dataset. Moreover, our augmentation policy for voxelized depth maps further enhances the accuracy of 3D hand pose estimation on real data. Our method produces visually more reasonable and realistic hand shapes on NYU and BigHand2.2M datasets compared to the existing approaches.

show abstract

Section: D Hand Shape Estimationmentioning

confidence: 99%

Section: D Hand Shape Estimationmentioning

confidence: 99%

Section: D Hand Shape Estimationmentioning

confidence: 99%

See 1 more Smart Citation

HandVoxNet: Deep Voxel-Based Network for 3D Hand Shape and Pose Estimation From a Single Depth Map

Malik

Abdelaziz

Elhayek

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In Lu et al (2019), and Li and Zhang (2019), supervised learning algorithms are defined for rigid registration, but with losses defined on dense correspondences between points, and on a soft-assigment matrix, respectively. Finally, Shimada et al (2019) designed a U-Net like architecture on voxel grids for non-rigid point set registration, however, their method is limited by the resolution of the grid and does not build latent representations of the scans, nor does it provide a morphable model. Abrevaya et al (2018) train a hybrid encoder-decoder architecture on rendered height maps from 3D face scans using an image CNN encoder and a multilinear decoder.…”

Section: Registrationmentioning

confidence: 99%

Shape My Face: Registering 3D Face Scans by Surface-to-Surface Translation

et al. 2021

View full text Add to dashboard Cite

Standard registration algorithms need to be independently applied to each surface to register, following careful pre-processing and hand-tuning. Recently, learning-based approaches have emerged that reduce the registration of new scans to running inference with a previously-trained model. The potential benefits are multifold: inference is typically orders of magnitude faster than solving a new instance of a difficult optimization problem, deep learning models can be made robust to noise and corruption, and the trained model may be re-used for other tasks, e.g. through transfer learning. In this paper, we cast the registration task as a surface-to-surface translation problem, and design a model to reliably capture the latent geometric information directly from raw 3D face scans. We introduce Shape-My-Face (SMF), a powerful encoder-decoder architecture based on an improved point cloud encoder, a novel visual attention mechanism, graph convolutional decoders with skip connections, and a specialized mouth model that we smoothly integrate with the mesh convolutions. Compared to the previous state-of-the-art learning algorithms for non-rigid registration of face scans, SMF only requires the raw data to be rigidly aligned (with scaling) with a pre-defined face template. Additionally, our model provides topologically-sound meshes with minimal supervision, offers faster training time, has orders of magnitude fewer trainable parameters, is more robust to noise, and can generalize to previously unseen datasets. We extensively evaluate the quality of our registrations on diverse data. We demonstrate the robustness and generalizability of our model with in-the-wild face scans across different modalities, sensor types, and resolutions. Finally, we show that, by learning to register scans, SMF produces a hybrid linear and non-linear morphable model. Manipulation of the latent space of SMF allows for shape generation, and morphing applications such as expression transfer in-the-wild. We train SMF on a dataset of human faces comprising 9 large-scale databases on commodity hardware.

show abstract

“…Multiple methods relax the rigidity constraints and recover general displacement fields between a template and a reference while imposing spatial smoothness on the displacement fields [8,20,21,3,25,23]. It is often assumed that the topology of template point sets cannot change, while non-isometric deformations including shrinkage and dilatations are allowed.…”

Section: Related Workmentioning

confidence: 99%

Optimising for Scale in Globally Multiply-Linked Gravitational Point Set Registration Leads to Singularities

Golyanik¹,

Theobalt²

2019

2019 International Conference on 3D Vision (3DV)

Self Cite

View full text Add to dashboard Cite

The recently introduced gravitational point set registration methods can robustly align point sets with high noise ratios. While many results with rotation and translation resolution have been presented in the literature so far, the uncertainty about the scale resolving capability of globally multiply-linked gravitational point set registration methods is remaining. To address the uncertainty, we analyse the gravitational potential energy (GPE) functional of the system with two point sets with multivariate calculus and come to the conclusion that GPE of a singularity, i.e., the state when the template collapses to a single point, always has a lower energy than the state of the optimal alignment. Moreover, the GPE as a function of scale monotonically increases and does not have equienergetic states, which we prove for several hollow and volumetric geometric primitives in R 2 and R 3 including a unit circle, a unit sphere, a unit disk and a unit ball. We perform a series of experiments with various regular shapes and different combinations of references and templates to validate our findings. The consequence is that in practice, it is highly unlikely for globally multiply-linked gravitational alignment to resolve the scale accurately. We propose ways to overcome the limitation in scale resolution, which can be considered in the next generation of gravitational approaches.

show abstract

DispVoxNets: Non-Rigid Point Set Alignment with Supervised Learning Proxies

Cited by 23 publications

References 62 publications

HandVoxNet: Deep Voxel-Based Network for 3D Hand Shape and Pose Estimation From a Single Depth Map

HandVoxNet: Deep Voxel-Based Network for 3D Hand Shape and Pose Estimation From a Single Depth Map

Shape My Face: Registering 3D Face Scans by Surface-to-Surface Translation

Optimising for Scale in Globally Multiply-Linked Gravitational Point Set Registration Leads to Singularities

Contact Info

Product

Resources

About