Bayesian Spatial Kernel Smoothing for Scalable Dense Semantic Mapping

Abstract

This paper develops a Bayesian continuous 3D semantic occupancy map from noisy point cloud measurements. In particular, we generalize the Bayesian kernel inference model for occupancy (binary) map building to semantic (multi-class) maps. The method nicely reverts to the original occupancy mapping framework when only one occupied class exists in obtained measurements. First, using Categorical likelihood and its conjugate prior distribution, we extend the counting sensor model for binary classification to a multi-class classification problem which results in a unified probabilistic model for both occupancy and semantic probabilities. Secondly, by applying a Bayesian spatial kernel inference to the semantic counting sensor model, we relax the independent grid assumption and bring smoothness and continuity to the map inference. These latter properties enable the method to exploit local correlations present in the environment to predict semantic probabilities in regions unobserved by the sensor while increasing the performance. Lastly, computational efficiency and scalability are achieved by leveraging sparse kernels and a test-data octrees data structure. The evaluations using multiple sequences of stereo camera and LiDAR datasets show that the proposed method consistently outperforms the compared baselines. We also present a qualitative evaluation using data collected by a biped robot platform on the University of Michigan -North Campus.

Learning Object Depth from Camera Motion and Video Object Segmentation

“…In the peach example (middle two rows), the detected object completely leaves view during the final two observations (9-10), providing no bounding box information at the prediction location. Finally, for the 16 mm die example (bottom), the camera starts far away from the small object (1), which is not detected until the camera is closer in the final observation (10).…”

Towards 3D LiDAR-based semantic scene understanding of 3D point cloud sequences: The SemanticKITTI Dataset

“…In addition to these benchmarks, we also see already adoption of the data for other domains benefiting from semantics, such as semantic SLAM (Chen et al, 2019;Gan et al, 2020), LiDAR-based localization (Yan et al, 2019), and loop closure detection (Chen et al, 2020). Moreover, domain adaptation (Jaritz et al, 2020;Langer et al, 2020) was also quite recently investigated to exploit our annotations for other sensors with different sensor geometries.…”

Improvements to Target-Based 3D LiDAR to Camera Calibration

“…The target is typically designed so that the vertices are easily distinguishable in the camera image. Denoting their corresponding coordinates in the image plane by {Y i } 4n i=1 completes the data required for the conceptual fitting problem in (1). While the cost to be minimized is nonlinear and non-convex, this is typically not a problem because CAD data can provide an adequate initial guess for local solvers, such as Levenberg-Marquardt; see [24] for example.…”

“…The desire to produce 3D-semantic maps [1] with our Cassie-series bipedal robot [2] has motivated us to fuse 3D-LiDAR and RGB-D monocular camera data for autonomous navigation [3]. Indeed, by mapping spatial LiDAR points onto a segmented and labeled camera image, one can associate the label of a pixel (or a region about it) to the LiDAR point as shown in Fig.…”

“…The lower right shows a single camera image; its segmentation with semantic labels is shown in the upper right. Synchronized LiDAR points are mapped onto the camera image, associated to a semantic label, and then re-projected to a local frame [25] along with the semantic labels to create a 3D semantically-labeled map [1], shown in the upper left. The sidewalk is marked in white, the grass in yellow-green, and trees in dark green.…”