Data Augmented 3D Semantic Scene Completion with 2D Segmentation Priors

Dourado, Aloisio; Guth, Frederico; Campos, Teófilo de

doi:10.1109/wacv51458.2022.00076

Cited by 11 publications

(5 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Semantic scene completion is achieved in [33][34][35][36][37][38][39][40][41][42][43][44][45][46][47][48][49][50] by adopting convolutions, transformers, or a combination of both, on different modalities like RGB images, depth images, and truncated signed distance functions (TSDFs), where the distance to the closest TSDF surface is computed at given 3D locations (usually voxel centers). Liu et al [33] proposed a disentangled framework, sequentially carrying out 2D semantic segmentation, 2D-3D projection, and 3D semantic scene completion.…”

Section: Multi-modality-based Methodsmentioning

confidence: 99%

Voxel- and Bird’s-Eye-View-Based Semantic Scene Completion for LiDAR Point Clouds

Liang,

Akhtar,

Vice

et al. 2024

Remote Sensing

View full text Add to dashboard Cite

Semantic scene completion is a crucial outdoor scene understanding task that has direct implications for technologies like autonomous driving and robotics. It compensates for unavoidable occlusions and partial measurements in LiDAR scans, which may otherwise cause catastrophic failures. Due to the inherent complexity of this task, existing methods generally rely on complex and computationally demanding scene completion models, which limits their practicality in downstream applications. Addressing this, we propose a novel integrated network that combines the strengths of 3D and 2D semantic scene completion techniques for efficient LiDAR point cloud scene completion. Our network leverages a newly devised lightweight multi-scale convolutional block (MSB) to efficiently aggregate multi-scale features, thereby improving the identification of small and distant objects. It further utilizes a layout-aware semantic block (LSB), developed to grasp the overall layout of the scene to precisely guide the reconstruction and recognition of features. Moreover, we also develop a feature fusion module (FFM) for effective interaction between the data derived from two disparate streams in our network, ensuring a robust and cohesive scene completion process. Extensive experiments with the popular SemanticKITTI dataset demonstrate that our method achieves highly competitive performance, with an mIoU of 35.7 and an IoU of 51.4. Notably, the proposed method achieves an mIoU improvement of 2.6 % compared to previous methods.

show abstract

Section: Multi-modality-based Methodsmentioning

confidence: 99%

Voxel- and Bird’s-Eye-View-Based Semantic Scene Completion for LiDAR Point Clouds

Liang,

Akhtar,

Vice

et al. 2024

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…A common requirement in applications is to infer semantic labels not only in directly observed but also in occluded space; to this end, semantic scene completion [34] seeks to address both scene occupancy completion and semantic object labeling jointly. Single-view depth images can be viewed as minimal input data [18, 34-36, 41, 42]; alternatives [2,12,14,22] (includ-ing our method) tackle completing fused reconstructions of entire 3D spaces. [12] jointly predicts a truncated, unsigned distance field and per-volume semantics in a series of hierarchy levels ranging from low to high resolution.…”

Section: Related Workmentioning

confidence: 99%

“…[12] jointly predicts a truncated, unsigned distance field and per-volume semantics in a series of hierarchy levels ranging from low to high resolution. Leveraging RGB-D image back-projection, [22] combines geometry and appearance features to infer semantics and refine 3D geometry; [14] adopts a view of RGB image segmentation as a prior and computes the final semantic scene completion using a 3D CNN. [2] exploits an interplay between the scene-and instance-level completion tasks and alternates between semantic scene completion and detection of object instances.…”

Section: Related Workmentioning

confidence: 99%

“…Moreover, even resorting to human experts for either scanning, semantic annotation, or artistic editing is unlikely to deliver flawless, complete digital 3D assets while being notoriously labourintensive [3,9]. This view has motivated much research on automatic (particularly, learning-based) approaches for completion, colorization, and semantic segmentation of raw single-view [2,14,34,36], multi-view [11,24], and fused 3D acquisitions [12,13,21,22].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

S4R: Self-Supervised Semantic Scene Reconstruction from RGB-D Scans

Artemorv¹,

Chen²,

Zhi³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Later, some followers (Guo and Tong 2018;Liu et al 2018;Garbade et al 2019;Li et al 2020b) leveraged the 2D semantic priors of color images via feature projection to improve the performance. Recent works (Dourado et al 2021;Dourado, Guth, and de Campos 2022;Wang, Lin, and Wan 2022) further fused complex features that are extracted from depth or color images to 2D semantic network. In addition, IMENet (Li, Ding, and Huang 2021) proposed an iterative fusion scheme to ensure the branches of 2D segmentation and 3D scene completion fully benefit each other.…”

Section: Related Workmentioning

confidence: 99%

CasFusionNet: A Cascaded Network for Point Cloud Semantic Scene Completion by Dense Feature Fusion

Tang

et al. 2023

AAAI

View full text Add to dashboard Cite

Semantic scene completion (SSC) aims to complete a partial 3D scene and predict its semantics simultaneously. Most existing works adopt the voxel representations, thus suffering from the growth of memory and computation cost as the voxel resolution increases. Though a few works attempt to solve SSC from the perspective of 3D point clouds, they have not fully exploited the correlation and complementarity between the two tasks of scene completion and semantic segmentation. In our work, we present CasFusionNet, a novel cascaded network for point cloud semantic scene completion by dense feature fusion. Specifically, we design (i) a global completion module (GCM) to produce an upsampled and completed but coarse point set, (ii) a semantic segmentation module (SSM) to predict the per-point semantic labels of the completed points generated by GCM, and (iii) a local refinement module (LRM) to further refine the coarse completed points and the associated labels from a local perspective. We organize the above three modules via dense feature fusion in each level, and cascade a total of four levels, where we also employ feature fusion between each level for sufficient information usage. Both quantitative and qualitative results on our compiled two point-based datasets validate the effectiveness and superiority of our CasFusionNet compared to state-of-the-art methods in terms of both scene completion and semantic segmentation. The codes and datasets are available at: https://github.com/JinfengX/CasFusionNet.

show abstract

Data Augmented 3D Semantic Scene Completion with 2D Segmentation Priors

Cited by 11 publications

References 31 publications

Voxel- and Bird’s-Eye-View-Based Semantic Scene Completion for LiDAR Point Clouds

Voxel- and Bird’s-Eye-View-Based Semantic Scene Completion for LiDAR Point Clouds

S4R: Self-Supervised Semantic Scene Reconstruction from RGB-D Scans

CasFusionNet: A Cascaded Network for Point Cloud Semantic Scene Completion by Dense Feature Fusion

Contact Info

Product

Resources

About