3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions

Zeng, Andy; Song, Shuran; NieBner, Matthias; Fisher, Matthew; Xiao, Jianxiong; Funkhouser, Thomas

doi:10.1109/cvpr.2017.29

Cited by 944 publications

(898 citation statements)

References 39 publications

Supporting

Mentioning

892

Contrasting

Unclassified

Order By: Relevance

“…Their operations are still on sparse volumes and not competent to the point‐wise features learning of large scale 3D points data due to lack of the abilities to be applied recursively for learning hierarchical features. The closest related to our work is 3DMatch [ZSN*17]. Zeng et al .…”

Section: Related Worksupporting

confidence: 58%

“…Zeng et al . [ZSN*17] learn a local volumetric patch descriptor for RGB‐D data. Although they show impressive results for aligning depth data for reconstruction, limited training data and limited resolution of voxel‐based surface neighbours still remain key challenges in this approach.…”

Section: Related Workmentioning

confidence: 99%

“…The key idea is to learn an effective feature presentation network to extract compact descriptors from local neighbours. For example, 3DMatch [ZSN*17] first presents a data‐driven model to learn the local volumetric patch descriptor to establish 3D data correspondence. This method uses the voxel data as input for learning local descriptors.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

SiamesePointNet: A Siamese Point Network Architecture for Learning 3D Shape Descriptor

Jian-hui

Wang

Mao

et al. 2019

Computer Graphics Forum

View full text Add to dashboard Cite

We present a novel deep learning approach to extract point‐wise descriptors directly on 3D shapes by introducing Siamese Point Networks, which contain a global shape constraint module and a feature transformation operator. Such geometric descriptor can be used in a variety of shape analysis problems such as 3D shape dense correspondence, key point matching and shape‐to‐scan matching. The descriptor is produced by a hierarchical encoder–decoder architecture that is trained to map geometrically and semantically similar points close to one another in descriptor space. Benefiting from the additional shape contrastive constraint and the hierarchical local operator, the learned descriptor is highly aware of both the global context and local context. In addition, a feature transformation operation is introduced in the end of our networks to transform the point features to a compact descriptor space. The feature transformation can make the descriptors extracted by our networks unaffected by geometric differences in shapes. Finally, an N‐tuple loss is used to train all the point descriptors on a complete 3D shape simultaneously to obtain point‐wise descriptors. The proposed Siamese Point Networks are robust to many types of perturbations such as the Gaussian noise and partial scan. In addition, we demonstrate that our approach improves state‐of‐the‐art results on the BHCP benchmark.

show abstract

Section: Related Worksupporting

confidence: 58%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

SiamesePointNet: A Siamese Point Network Architecture for Learning 3D Shape Descriptor

Jian-hui

Wang

Mao

et al. 2019

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…However, such descriptors either require a volumetric representation which can be very computationaly expensive e.g., Zeng et al (2017), or cannot capture local finegrained structures see e.g., Qi et al (2017). Furthermore, they are not intrinsically invariant to rotations and try to overcome this by augmenting the data with random rotations (usually only around the up-axis).…”

Section: Local Feature Descriptorsmentioning

confidence: 99%

Learned Compact Local Feature Descriptor for TLS-Based Geodetic Monitoring of Natural Outdoor Scenes

Gojčič¹,

Zhou²

2018

ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

ABSTRACT:The advantages of terrestrial laser scanning (TLS) for geodetic monitoring of man-made and natural objects are not yet fully exploited. Herein we address one of the open challenges by proposing feature-based methods for identification of corresponding points in point clouds of two or more epochs. We propose a learned compact feature descriptor tailored for point clouds of natural outdoor scenes obtained using TLS. We evaluate our method both on a benchmark data set and on a specially acquired outdoor dataset resembling a simplified monitoring scenario where we successfully estimate 3D displacement vectors of a rock that has been displaced between the scans. We show that the proposed descriptor has the capacity to generalize to unseen data and achieves state-of-the-art performance while being time efficient at the matching step due the low dimension.

show abstract

“…Newcombe et al [2011) present a technique for real-time reconstruction where sensor data is continuously received and fused into a 3D volume. Such fusionbased reconstructions, however, still suffer from drift issues due to ICP registration euor, in particular in larger environments [Nießner et al 2013 J. Advancedmethods, such as bunclle adjustment [Agarwal et aL 2010), robust optimization (Choi et al 2015), structure-based alignment [Zhang et al 2014), or feature matching with deep learning [Zeng et aL 2016] are generally expensive for online use. This makes the optimization of smooth sensor trajectories especially important for autonomous online reconstruction.…”

Section: Related Workmentioning

confidence: 99%

Autonomous reconstruction of unknown indoor scenes guided by time-varying tensor fields

Zheng

Yan

et al. 2017

ACM Trans. Graph.

Self Cite

View full text Add to dashboard Cite

(a) (b) (c) (d) Fig. 1. We introduce an algorithm for the autonomous reconstruction of indoor scenes, based on time-varying tensor fields (a). Given the partially scanned scene, a 2D tensor field is computed on-the-fly over the floor plane, constrained by the partial reconstruction (b). The robot is guided by the field with smooth paths, which are locally formed with field advection (red curve in (a)) and globally planned with the help of the field topology (see the curve networks in (b) and (c), with reconstruction uncertainty color-coded along the curves). The topological structure is well-defined for incomplete scenes (b), suited for guiding exploratory reconstruction of unknown scenes. The output is a full 3D reconstruction (d), at which the topology of the field reflects the scene layout (c).Autonomous reconstruction of unknown scenes by a mobile robot inherently poses the question of balancing between exploration efficacy and reconstruction quality. We present a navigation-by-reconstruction approach to address this question, where moving paths of the robot are planned to account for both global efficiency for fast exploration and local smoothness to obtain high-quality scans. An RGB-D camera, attached to the robot arm, is dictated by the desired reconstruction quality as well as the movement of the robot itself. Our key idea is to harness a time-varying tensor field to guide robot movement, and then solve for 3D camera control under the constraint of the 2D robot moving path. The tensor field is updated in real time, conforming to the progressively reconstructed scene. We show that tensor fields are well suited for guiding autonomous scanning for two reasons: first, they contain sparse and controllable singularities that allow generating a locally smooth robot path, and second, their topological structure can be used for globally efficient path routing within a partially reconstructed scene. We have conducted numerous tests with a mobile robot, and demonstrate that our method leads to a smooth exploration and high-quality reconstruction of unknown indoor scenes.

show abstract

3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions

Cited by 944 publications

References 39 publications

SiamesePointNet: A Siamese Point Network Architecture for Learning 3D Shape Descriptor

SiamesePointNet: A Siamese Point Network Architecture for Learning 3D Shape Descriptor

Learned Compact Local Feature Descriptor for TLS-Based Geodetic Monitoring of Natural Outdoor Scenes

Autonomous reconstruction of unknown indoor scenes guided by time-varying tensor fields

Contact Info

Product

Resources

About