CF-SIS: Semantic-Instance Segmentation of 3D Point Clouds by Context Fusion with Self-Attention

Wen, Xin; Han, Zhizhong; Youk, Geunhyuk; Liu, Yu‐Shen

doi:10.1145/3394171.3413829

Cited by 40 publications

(16 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PointNet [32] and Point-Net++ [33] are the pioneer works to understand this kind of irregular data. After that, lots of studies [27,39] are proposed to improve the interpretability of network for point clouds in different tasks, such as segmentation [28,29,37], classification [28,29,37], reconstruction [9,11,14,18], completion [15,16,38]. Besides, the learned deep features of a single point or the whole shape could also be applied to 3D shape based cross-modal applications, for example, shape-to-text matching in our case.…”

Section: Point-based 3d Deep Learningmentioning

confidence: 99%

Part2Word: Learning Joint Embedding of Point Clouds and Text by Matching Parts to Words

Tang¹,

Yang²,

Wu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

It is important to learn joint embedding for 3D shapes and text in different shape understanding tasks, such as shape-text matching, retrieval, and shape captioning. Current multi-view based methods learn a mapping from multiple rendered views to text. However, these methods can not analyze 3D shapes well due to the selfocclusion and limitation of learning manifolds. To resolve this issue, we propose a method to learn joint embedding of point clouds and text by matching parts from shapes to words from sentences in a common space. Specifically, we first learn segmentation prior to segment point clouds into parts. Then, we map parts and words into an optimized space, where the parts and words can be matched with each other. In the optimized space, we represent a part by aggregating features of all points within the part, while representing each word with its context information, where we train our network to minimize the triplet ranking loss. Moreover, we also introduce cross-modal attention to capture the relationship of part-word in this matching procedure, which enhances joint embedding learning. Our experimental results outperform the state-of-the-art in multimodal retrieval under the widely used benchmark. CCS CONCEPTS• Computing methodologies → Visual content-based indexing and retrieval; Learning latent representations.

show abstract

Section: Point-based 3d Deep Learningmentioning

confidence: 99%

Part2Word: Learning Joint Embedding of Point Clouds and Text by Matching Parts to Words

Tang¹,

Yang²,

Wu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The mIoU is obtained by averaging the IoU of all shapes. We compare our model against the state-of-theart point-based methods [14,28,18,51,37,34,11,30,47,3,54,52,20,19,21,49,22], voxel-based methods [6] and the newest point-voxel-based model [23]. To better balance the trade-off between time efficiency and accuracy, we also reduce the output feature channels to 50% and 25%, and marked as MVPCNN (0.5×Ch) and MVPCNN (0.25×Ch) respectively.…”

Section: Part Segmentationmentioning

confidence: 99%

“…Apart from mIoU, mean accuracy (mAcc) is also used to evaluate the performance of our proposed model. In addition to comparing with the state-of-the-art pointbased [28,47,20,11,30,40,56,42,19,53,21,49] and voxel-based methods [6], we also compare with the newest point-voxel-based model [23]. Table 2 shows the results of all methods on S3DIS dataset, and Figure 4…”

Section: Part Segmentationmentioning

confidence: 99%

Multi Voxel-Point Neurons Convolution (MVPConv) for Fast and Accurate 3D Deep Learning

Zhou,

Cao,

Zhang

et al. 2021

Preprint

View full text Add to dashboard Cite

We present a new convolutional neural network, called Multi Voxel-Point Neurons Convolution (MVPConv), for fast and accurate 3D deep learning. The previous works adopt either individual point-based features or local-neighboring voxel-based features to process 3D model, which limits the performance of models due to the inefficient computation. Moreover, most of the existing 3D deep learning frameworks aim at solving one specific task, and only a few of them can handle a variety of tasks. Integrating both the advantages of the voxel and point-based methods, the proposed MVPConv can effectively increase the neighboring collection between point-based features and also promote the independence among voxel-based features. Simply replacing the corresponding convolution module with MVPConv, we show that MVPConv can fit in different backbones to solve a wide range of 3D tasks. Extensive experiments on benchmark datasets such as ShapeNet Part, S3DIS and KITTI for various tasks show that MVPConv improves the accuracy of the backbone (PointNet) by up to 36%, and achieves higher accuracy than the voxel-based model with up to 34× speedup. In addition, MVPConv also outperforms the state-of-the-art point-based models with up to 8× speedup. Notably, our MVPConv achieves better accuracy than the newest point-voxel-based model PVCNN (a model more efficient than PointNet) with lower latency.

show abstract

“…Deep learning models have been playing an important role in different 3D computer vision applications [54,49,52,65,27,17,60,4,58,28,3,50,18,23,21,24,25,29,30,20,32,26,22,47,46,31,34,33,19,64]. In the following, we will briefly review work related to learning implicit functions for 3D shapes in different ways.…”

Section: Related Workmentioning

confidence: 99%

Neural-Pull: Learning Signed Distance Functions from Point Clouds by Learning to Pull Space onto Surfaces

Ma,

Han,

Liu

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Reconstructing continuous surfaces from 3D point clouds is a fundamental operation in 3D geometry processing. Several recent state-of-the-art methods address this problem using neural networks to learn signed distance functions (SDFs). In this paper, we introduce Neural-Pull, a new approach that is simple and leads to high quality SDFs. Specifically, we train a neural network to pull query 3D locations to their closest neighbors on the surface using the predicted signed distance values and the gradient at the query locations, both of which are computed by the network itself. The pulling operation moves each query location with a stride given by the distance predicted by the network. Based on the sign of the distance, this may move the query location along or against the direction of the gradient of the SDF. This is a differentiable operation that allows us to update the signed distance value and the gradient simultaneously during training. Our outperforming results under widely used benchmarks demonstrate that we can learn SDFs more accurately and flexibly for surface reconstruction and single image reconstruction than the state-of-theart methods.

show abstract

CF-SIS: Semantic-Instance Segmentation of 3D Point Clouds by Context Fusion with Self-Attention

Abstract: 3D Semantic-Instance Segmentation (SIS) is a newly emerging research direction that aims to understand visual information of 3D scene on both semantic and instance level. The main difficulty lies in how to coordinate the paradox between mutual aid and suboptimal problem.

Cited by 40 publications

References 35 publications

Part2Word: Learning Joint Embedding of Point Clouds and Text by Matching Parts to Words

Part2Word: Learning Joint Embedding of Point Clouds and Text by Matching Parts to Words

Multi Voxel-Point Neurons Convolution (MVPConv) for Fast and Accurate 3D Deep Learning

Neural-Pull: Learning Signed Distance Functions from Point Clouds by Learning to Pull Space onto Surfaces

Contact Info

Product

Resources

About