Zero-shot Learning of 3D Point Cloud Objects

Abstract

Recent deep learning architectures can recognize instances of 3D point cloud objects of previously seen classes quite well. At the same time, current 3D depth camera technology allows generating/segmenting a large amount of 3D point cloud objects from an arbitrary scene, for which there is no previously seen training data. A challenge for a 3D point cloud recognition system is, then, to classify objects from new, unseen, classes. This issue can be resolved by adopting a zero-shot learning (ZSL) approach for 3D data, similar to the 2D image version of the same problem. ZSL attempts to classify unseen objects by comparing semantic information (attribute/word vector) of seen and unseen classes. Here, we adapt several recent 3D point cloud recognition systems to the ZSL setting with some changes to their architectures. To the best of our knowledge, this is the first attempt to classify unseen 3D point cloud objects in the ZSL setting. A standard protocol (which includes the choice of datasets and the seen/unseen split) to evaluate such systems is also proposed. Baseline performances are reported using the new protocol on the investigated models. This investigation throws a new challenge to the 3D point cloud recognition community that may instigate numerous future works.

Deep Learning on Point Clouds and Its Application: A Survey

“…A promising solution is to address the raw point clouds with the ConvNets. Since ConvNets has the advantage of overlapping during convolutional operation [115,116,117], it may benefit the future architecture of deep learning models for the point cloud to take the characteristics, i.e., interaction among points, into consideration. Usually, ConvNets are used to extract multi-scale semantic features.…”

“…Finally, zero-shot learning [115] is also an exciting topic for deep learning models directly processing raw point clouds. After obtaining the feature maps, it uses a semantic embedding for applications such as object detection.…”

Transfer Learning in urban object classification: Online images to recognize point clouds

“…Accuracy in the best of their case studies reached 86.8%. Some authors modify PointNet to achieve improvements, for example, through zero-shot learning [61]. As with 3D-CNN, some authors choose to merge point clouds with multi-view images as input data, as in PVNet [62] and PVRNet [63].…”

Classification and Segmentation of Mining Area Objects in Large-Scale Spares Lidar Point Cloud Using a Novel Rotated Density Network

“…To overcome it, early researchers used volumetric [20][21][22], multi-view [6,[23][24][25], or other feature representations in order to first build 3D models. In recent years, the trend has shifted instead to using raw 3D data directly [26][27][28][29][30][31]. Therefore, we divided these approaches into two groups, coordinate-based and feature-based network.…”