TWIST: Two-Way Inter-label Self-Training for Semi-supervised 3D Instance Segmentation

Chu, Ruihang; Ye, Xiaoqing; Liu, Zhengzhe; Tan, Xiao; Qi, Xiaojuan; Jia, Jiaya

doi:10.1109/cvpr52688.2022.00117

Cited by 24 publications

(8 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sparse Convolutional Networks Sparse CNNs become mainframe backbone networks in 3D deep learning [10,11,23,41] for its efficiency. It is common wisdom that its representation ability is limited for prediction.…”

Section: Efficiency On Argoverse2mentioning

confidence: 99%

VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking

Chen¹,

Liu²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

3D object detectors usually rely on hand-crafted proxies, e.g., anchors or centers, and translate well-studied 2D frameworks to 3D. Thus, sparse voxel features need to be densified and processed by dense prediction heads, which inevitably costs extra computation. In this paper, we instead propose VoxelNext for fully sparse 3D object detection. Our core insight is to predict objects directly based on sparse voxel features, without relying on hand-crafted proxies. Our strong sparse convolutional network Vox-elNeXt detects and tracks 3D objects through voxel features entirely. It is an elegant and efficient framework, with no need for sparse-to-dense conversion or NMS postprocessing. Our method achieves a better speed-accuracy trade-off than other mainframe detectors on the nuScenes dataset. For the first time, we show that a fully sparse voxelbased representation works decently for LIDAR 3D object detection and tracking. Extensive experiments on nuScenes, Waymo, and Argoverse2 benchmarks validate the effectiveness of our approach. Without bells and whistles, our model outperforms all existing LIDAR methods on the nuScenes tracking test benchmark. Code and models are available at github.com/dvlab-research/VoxelNeXt.

show abstract

Section: Efficiency On Argoverse2mentioning

confidence: 99%

VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking

Chen¹,

Liu²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Different from 3D instance segmentation [45,7], we study the higher-order problem of object relationship prediction in the scene. Note that we use point cloud data with real instance indexes but without category labels.…”

Section: Scene Graph Initializationmentioning

confidence: 99%

“…(1, 2) 2 initialize semantic knowledge Z emb from BERT 3 for each node X Vi ∈ X V do 4 update X Vi and X E (i,j) using Eq. (5,6,7,8,9,10) 5 end 6 update X with semantic knowledge using Eq. (13,14,15) 7 for l in L do 8 update X Vi and X E (i,j) using Eq.…”

Section: Analysis Of Sil Locationmentioning

confidence: 99%

See 1 more Smart Citation

Revisiting Transformer for Point Cloud-based 3D Scene Graph Generation

Lv¹,

Qi²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

In this paper, we propose the semantic graph Transformer (SGT) for the 3D scene graph generation. The task aims to parse a cloud point-based scene into a semantic structural graph, with the core challenge of modeling the complex global structure. Existing methods based on graph convolutional networks (GCNs) suffer from the oversmoothing dilemma and could only propagate information from limited neighboring nodes. In contrast, our SGT uses Transformer layers as the base building block to allow global information passing, with two types of proposed Transformer layers tailored for the 3D scene graph generation task. Specifically, we introduce the graph embedding layer to best utilize the global information in graph edges while maintaining comparable computation costs. Additionally, we propose the semantic injection layer to leverage categorical text labels and visual object knowledge. We benchmark our SGT on the established 3DSSG benchmark and achieve a 35.9% absolute improvement in relationship prediction's R@50 and an 80.4% boost on the subset with complex scenes over the state-of-the-art. Our analyses further show SGT's superiority in the long-tailed and zero-shot scenarios. We will release the code and model.

show abstract

“…However, they all rely on full supervision. Recently, TWIST [8] also employs self-training-based approach to conduct effective semi-supervised learning. They innovatively proposed novel proposal re-correction module to filter out the low-quality proposals and enhance the pseudo label quality.…”

Section: D Semantic/instance Segmentation and Object Detectionmentioning

confidence: 99%

RM3D: Robust Data-Efficient 3D Scene Parsing via Traditional and Learnt 3D Descriptors-Based Semantic Region Merging

Liu

2022

Int J Comput Vis

View full text Add to dashboard Cite

Existing state-of-the-art 3D point cloud understanding methods merely perform well in a fully supervised manner. To the best of our knowledge, there exists no unified framework that simultaneously solves the downstream high-level understanding tasks including both segmentation and detection, especially when labels are extremely limited. This work presents a general and simple framework to tackle point cloud understanding when labels are limited. The first contribution is that we have done extensive methodology comparisons of traditional and learned 3D descriptors for the task of weakly supervised 3D scene understanding, and validated that our adapted traditional PFH-based 3D descriptors show excellent generalization ability across different domains. The second contribution is that we proposed a learning-based region merging strategy based on the affinity provided by both the traditional/learned 3D descriptors and learned semantics. The merging process takes both low-level geometric and high-level semantic feature correlations into consideration. Experimental results demonstrate that our framework has the best performance among the three most important weakly supervised point clouds understanding tasks including semantic segmentation, instance segmentation, and object detection even when very limited number of points are labeled. Our method, termed Region Merging 3D (RM3D), has superior performance on Scan-Net data-efficient learning online benchmarks and other four large-scale 3D understanding benchmarks under various experimental settings, outperforming current arts by a margin for various 3D understanding tasks without complicated learning strategies such as active learning.

show abstract

TWIST: Two-Way Inter-label Self-Training for Semi-supervised 3D Instance Segmentation

Cited by 24 publications

References 37 publications

VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking

VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking

Revisiting Transformer for Point Cloud-based 3D Scene Graph Generation

RM3D: Robust Data-Efficient 3D Scene Parsing via Traditional and Learnt 3D Descriptors-Based Semantic Region Merging

Contact Info

Product

Resources

About