Consensus of Non-rigid Reconstructions

Lee, Minsik; Cho, Jungchan; Oh, Songhwai

doi:10.1109/cvpr.2016.505

Cited by 47 publications

(85 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, we proposed to use the coherence of the final dictionary as a model quality measure, offering a practical way to avoid over-fitting and select the best checkpoint during training without relying on 3D ground-truth. [20], SPS [18], NLO [12]. Each column corresponds to reconstructions of a certain frame, randomly selected from each subject.…”

Section: Resultsmentioning

confidence: 99%

“…Moreover, our network also show a well generalization to unseen data which improve the effectiveness in real world applications. For qualitative evaluation, we randomly select a frame from Subjects 01 05 18 23 64 70 102 106 123 127 Average Relative CNS [20] each subject and render the reconstructed human skeleton in Figure 5. This visually verifies the impressive performance of our deep solution.…”

Section: Large-scale Nrsf M On Cmu Mocapmentioning

confidence: 99%

See 1 more Smart Citation

Deep Non-Rigid Structure From Motion

Kong

Lucey

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Non-Rigid Structure from Motion (NRSfM) refers to the problem of reconstructing cameras and the 3D point cloud of a non-rigid object from an ensemble of images with 2D correspondences. Current NRSfM algorithms are limited from two perspectives: (i) the number of images, and (ii) the type of shape variability they can handle. These difficulties stem from the inherent conflict between the condition of the system and the degrees of freedom needing to be modeled -which has hampered its practical utility for many applications within vision. In this paper we propose a novel hierarchical sparse coding model for NRSFM which can overcome (i) and (ii) to such an extent, that NRSFM can be applied to problems in vision previously thought too ill posed. Our approach is realized in practice as the training of an unsupervised deep neural network (DNN) auto-encoder with a unique architecture that is able to disentangle pose from 3D structure. Using modern deep learning computational platforms allows us to solve NRSfM problems at an unprecedented scale and shape complexity. Our approach has no 3D supervision, relying solely on 2D point correspondences. Further, our approach is also able to handle missing/occluded 2D points without the need for matrix completion. Extensive experiments demonstrate the impressive performance of our approach where we exhibit superior precision and robustness against all available state-of-the-art works in some instances by an order of magnitude. We further propose a new quality measure (based on the network weights) which circumvents the need for 3D ground-truth to ascertain the confidence we have in the reconstructability. We believe our work to be a significant advance over state-of-the-art in NRSFM.Index Terms-Nonrigid structure from motion, hierarchical sparse coding, deep neural network, reconstructability, missing data. ! • C. Kong and S. Lucey are with the

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Large-scale Nrsf M On Cmu Mocapmentioning

confidence: 99%

Deep Non-Rigid Structure From Motion

Kong

Lucey

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

show abstract

“…Deep-NRSfM Weaksup-bs Ours GT. P-MPJPE MPJPE depth error Ranklet [11] 281.1 --Sparse [20] 217.4 --SPM(2k) [9] 209.5 --SFC [22] 167.1 218.0 135.6 KSTA(5k) [16] 123.6 --RIKS(5k) [17] 103.9 --Consensus [25] 79.6 120. 3D pose estimation network: We select the integral regression network [36] due to its state-of-the-art performance in human pose estimation.…”

Section: Consensusmentioning

confidence: 99%

Distill Knowledge From NRSfM for Weakly Supervised 3D Pose Learning

Wang

Kong

Lucey

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

We propose to learn a 3D pose estimator by distilling knowledge from Non-Rigid Structure from Motion (NRSfM). Our method uses solely 2D landmark annotations. No 3D data, multi-view/temporal footage, or object specific prior is required. This alleviates the data bottleneck, which is one of the major concern for supervised methods. The challenge for using NRSfM as teacher is that they often make poor depth reconstruction when the 2D projections have strong ambiguity. Directly using those wrong depth as hard target would negatively impact the student. Instead, we propose a novel loss that ties depth prediction to the cost function used in NRSfM. This gives the student pose estimator freedom to reduce depth error by associating with image features. Validated on H3.6M dataset, our learned 3D pose estimation network achieves more accurate reconstruction compared to NRSfM methods. It also outperforms other weakly supervised methods, in spite of using significantly less supervision.

show abstract

“…Evaluation on CMU sequences with two subjects. The table reports the 3D reconstruction error eX for the following NRSfM baselines considering 2D tracks without missing data: CSF [21], KSTA [22], SPM [16], EM-PND [24], TUS [40], GBNR [18] and CNR [25]; and ours. For our approach, we also show the clustering errors eS and eT , where we include the number of spatial and temporal clusters in brackets.…”

Section: Spatial and Temporal Clusteringmentioning

confidence: 99%

“…σ f x , σ f y and σ f z are the error standard deviations at frame f . We compare the reconstruction accuracy of our approach, denoted DUST, against seven NRSfM baselines: the trajectory-space methods CSF [21] and KSTA [22]; the block matrix approach BMM [16], the probabilisticnormal-distribution method EM-PND [24], the temporal union of subspaces TUS [40], the grouping-based NRSfM of GBNR [18] and the consensus NRSfM of CNR [25]. For CSF [21] and KSTA [22], we manually set the rank of the subspace to the value yielding the best results.…”

Section: Experimental Evaluationmentioning

confidence: 99%

DUST: Dual Union of Spatio-Temporal Subspaces for Monocular Multiple Object 3D Reconstruction

Agudo

Moreno-Noguer

2017

2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

We present an approach to reconstruct the 3D shape of multiple deforming objects from incomplete 2D trajectories acquired by a single camera. Additionally, we simultaneously provide spatial segmentation (i.e., we identify each of the objects in every frame) and temporal clustering (i.e., we split the sequence into primitive actions). This advances existing work, which only tackled the problem for one single object and non-occluded tracks. In order to handle several objects at a time from partial observations, we model point trajectories as a union of spatial and temporal subspaces, and optimize the parameters of both modalities, the nonobserved point tracks and the 3D shape via augmented Lagrange multipliers. The algorithm is fully unsupervised and results in a formulation which does not need initialization. We thoroughly validate the method on challenging scenarios with several human subjects performing different activities which involve complex motions and close interaction. We show our approach achieves state-of-the-art 3D reconstruction results, while it also provides spatial and temporal segmentation.

show abstract

Consensus of Non-rigid Reconstructions

Cited by 47 publications

References 25 publications

Deep Non-Rigid Structure From Motion

Deep Non-Rigid Structure From Motion

Distill Knowledge From NRSfM for Weakly Supervised 3D Pose Learning

DUST: Dual Union of Spatio-Temporal Subspaces for Monocular Multiple Object 3D Reconstruction

Contact Info

Product

Resources

About