CasFusionNet: A Cascaded Network for Point Cloud Semantic Scene Completion by Dense Feature Fusion

Abstract

Semantic scene completion (SSC) aims to complete a partial 3D scene and predict its semantics simultaneously. Most existing works adopt the voxel representations, thus suffering from the growth of memory and computation cost as the voxel resolution increases. Though a few works attempt to solve SSC from the perspective of 3D point clouds, they have not fully exploited the correlation and complementarity between the two tasks of scene completion and semantic segmentation. In our work, we present CasFusionNet, a novel cascaded network for point cloud semantic scene completion by dense feature fusion. Specifically, we design (i) a global completion module (GCM) to produce an upsampled and completed but coarse point set, (ii) a semantic segmentation module (SSM) to predict the per-point semantic labels of the completed points generated by GCM, and (iii) a local refinement module (LRM) to further refine the coarse completed points and the associated labels from a local perspective. We organize the above three modules via dense feature fusion in each level, and cascade a total of four levels, where we also employ feature fusion between each level for sufficient information usage. Both quantitative and qualitative results on our compiled two point-based datasets validate the effectiveness and superiority of our CasFusionNet compared to state-of-the-art methods in terms of both scene completion and semantic segmentation. The codes and datasets are available at: https://github.com/JinfengX/CasFusionNet.

Voxel- and Bird’s-Eye-View-Based Semantic Scene Completion for LiDAR Point Clouds

“…These methods [2,[21][22][23][24][25][26][27][28] directly apply convolution or transformer operations to original point clouds, leveraging their distance measurements and fine semantic descriptions for semantic scene completion. Roldao et al [2] introduced LMSCNet, which uses lighter 2D convolutions to process point clouds, thus reducing the heavy computation caused by voxelization.…”

“…Furthermore, Xiong et al [26] proposed UltraLiDAR, which uses a sparse-to-dense data reconstruction pipeline to enhance data density and a zero-shot scheme to improve the generalization ability of the trained detection models. Xu et al [27] presented CasFusionNet, deploying the following: (i) a global completion module (GCM) to produce an unsampled and completed but coarse point set, (ii) a semantic segmentation module (SSM) to predict the per-point semantic labels for the completed points, and (iii) a local refinement module (LRM), which further refines the coarse completed points and the associated labels. While point-based methods can effectively process point clouds, their high computational load and unstructured data processing remain open challenges.…”