NNNet: New Normal Guided Depth Completion From Sparse LiDAR Data and Single Color Image

Abstract

In this paper, we propose new normal guided depth completion from sparse LiDAR data and single color image, named NNNet. Sparse depth completion often uses normal maps as a constraint for model training. However, direct construction of a normal map from the color image causes a lot of noise in the normal map and reduces the model performance. Thus, we use a new normal map as an intermediate constraint to promote the fusion of multi-modal features. We generate the new normal map from the sparse LiDAR depth data to use it as a constraint for network training. The new normal map is generated by converting the input depth into a grayscale image, constructing a normal map, replacing the Z channel of the normal map with the original depth, and finally adding a mask. Based on the new normal map, we construct an end-to-end network NNNet for sparse depth completion guided by its corresponding color image. NNNet consists of two branches. The one branch generates the new normal map from the depth image and its corresponding color image, while the other branch constructs a dense depth image from the sparse depth and the predicted new normal map. The two branches fully merge the features through skip connection. In loss function, we use L2 loss to ensure that the new normal map plays a restrictive role. Finally, we generate the dense depth image by refining it with a spatial propagation network. Experimental results show that the new normal map provides effective constraints for sparse depth completion. Moreover, NNNet achieves 724.14 in terms of RMSE and outperforms most of the current state-of-the-art methods.

Sensing and Artificial Perception for Robots in Precision Forestry: A Survey

“…TUPPer-Map [205] RGB-D Urban Mesh Truncated Signed Distance Field (TSDF) Kimera-Multi [206] RGB-D/PCL Urban Mesh TSDF SSC [207] PCL Urban 3D Points Point Cloud MultiLayerMapping [208] RGB-D Urban Voxels Multi-Layered BGKOctoMap [211] Semantic OctoMap [31] RGB-D/PCL Forest/Urban Voxels OctoMap [210] Active MS Mapping [202,209] Stereo/PCL Forest/Urban 3D Models Factor-Graph…”

“…There are only two methods that we found that address forestry applications, the first of which was proposed by Liu et al [202] and Liu and Jung [209], and the second by Russell et al [31]. The first group of authors present a comprehensive framework for large-scale autonomous flight with real-time semantic SLAM under dense forest canopies.…”

“…We analysed the implementation aspect of each method in terms of the environments used, and the geometry and data structure for its spatial representation, as there are few works that have been tested or compared under similar conditions. These particular features are important since they provide an indication of how each technique would perform (particularly in terms of the usage of computational resources) in unstructured, large, outdoor environments, such as woodlands (note that only the active metric-semantic mapping technique by Liu et al [202], Liu and Jung [209] and the Semantic OctoMap solution by Russell et al [31] are specifically reported to have been used in this type of environment).…”