ManhattanSLAM: Robust Planar Tracking and Mapping Leveraging Mixture of Manhattan Frames

Abstract

In this paper, a robust RGB-D SLAM system is proposed to utilize the structural information in indoor scenes, allowing for accurate tracking and efficient dense mapping on a CPU. Prior works have used the Manhattan World (MW) assumption to estimate low-drift camera pose, in turn limiting the applications of such systems. This paper, in contrast, proposes a novel approach delivering robust tracking in MW and non-MW environments. We check orthogonal relations between planes to directly detect Manhattan Frames, modeling the scene as a Mixture of Manhattan Frames. For MW scenes, we decouple pose estimation and provide a novel drift-free rotation estimation based on Manhattan Frame observations. For translation estimation in MW scenes and full camera pose estimation in non-MW scenes, we make use of point, line and plane features for robust tracking in challenging scenes. Additionally, by exploiting plane features detected in each frame, we also propose an efficient surfel-based dense mapping strategy, which divides each image into planar and non-planar regions. Planar surfels are initialized directly from sparse planes in our map while non-planar surfels are built by extracting superpixels. We evaluate our method on public benchmarks for pose estimation, drift and reconstruction accuracy, achieving superior performance compared to other state-of-the-art methods. We will open-source our code in the future.

MSC-VO: Exploiting Manhattan and Structural Constraints for Visual Odometry

“…Despite its advantages, this method can not reduce the long-term rotation error as the MW assumption does. Another solution is proposed in [13], where the authors use either a decoupled or a non-decoupled tracking strategy depending on whether the scene meets the MW assumption. These strategies permit these works to not only focus on a specific environment.…”

“…This assumption is fundamentally used during the tracking stage [8]- [12]. Nonetheless, these methods do not usually take into account that some indoor environments are not strictly adhering to this assumption, leading to degradation in accuracy or even to tracking failures [13]. Additionally, most of these works rely on planes to estimate and track the MA.…”

Semantic Dense Reconstruction with Consistent Scene Segments

“…Based on RGB-D images, tracking and mapping methods [25] are used to build global 3D maps that are bridges for complete scene understanding systems. Multi-featurebased trackers [17], [18] achieve robust estimations in indoor scenes, but those mapping parts aim to maintain sparse features for removing camera pose drift rather than reconstructing dense maps. Different from those methods, KinectFision [26] and BundleFusion [27] focus their mind on dense reconstruction by using GPUs to obtain on-the-fly 3D scene reconstruction.…”

“…Compared to the existing 3D segmentation networks [3], [9] and scene graph generation approach [15], our dense semantic reconstruction method has a more complete function that adopts continuous RGB-D frames and outputs dense semantic 3D models. Based on 2D segmentation algorithms [6], [7] and our camera trackers [17], [18], we maintain a semantic sparse map that saves the probability of each object. Since semantic predictions from partial views are not as reliable as full ones', the semantic sparse map is used to correct wrong 2D segments existed in bad cases.…”

PlaneRecNet: Multi-Task Learning with Cross-Task Consistency for Piece-Wise Plane Detection and Reconstruction from a Single RGB Image

“…In indoor environments, orthogonal or parallel planar regions (walls, floor and ceiling) can form a layout that satisfies the Manhattan World (MW) assumption. Some SLAM systems [2,6,14,15,30,33,46] employ such structural features for fast dense map reconstruction or low drift localization. On the other hand, planar surface representations are also utilized to solve the occlusionawareness in indoor scene understanding [11].…”