A comparative study of data fusion for RGB-D based visual recognition

Sanchez-Riera, Jordi; Hua, Kai-Lung; Hsiao, Yuan-Sheng; Lim, Theam Soon; Hidayati, Shintami Chusnul; Cheng, Winton

doi:10.1016/j.patrec.2015.12.006

Cited by 59 publications

(26 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The multimodal fusion research communities have achieved substantial advances [49]. The fusion of different modalities is generally performed at two levels, namely early level (or feature level) fusion and late level (or decision level) fusion [50], [52], [60]. (1) Early level fusion In the early fusion approach, the RGB and depth features are extracted from two separate CNNs.…”

Section: ) Multimodal Fusion Strategiesmentioning

confidence: 99%

“…This paper focuses on the RGB and depth modalities which can be considered as complete multimodal data. In the literature, early fusion and late fusion are the two most popular fusion schemes [50], [52], [60] in complete multimodal data fusion. Early fusion, also known as feature fusion, integrates data from different modalities before being passed to a classifier.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

RGB-D-Based Object Recognition Using Multimodal Convolutional Neural Networks: A Survey

et al. 2019

View full text Add to dashboard Cite

Object recognition in real-world environments is one of the fundamental and key tasks in computer vision and robotics communities. With the advanced sensing technologies and low-cost depth sensors, the high-quality RGB and depth images can be recorded synchronously, and the object recognition performance can be improved by jointly exploiting them. RGB-D-based object recognition has evolved from early methods that using hand-crafted representations to the current state-of-the-art deep learning-based methods. With the undeniable success of deep learning, especially convolutional neural networks (CNNs) in the visual domain, the natural progression of deep learning research points to problems involving larger and more complex multimodal data. In this paper, we provide a comprehensive survey of recent multimodal CNNs (MMCNNs)-based approaches that have demonstrated significant improvements over previous methods. We highlight two key issues, namely, training data deficiency and multimodal fusion. In addition, we summarize and discuss the publicly available RGB-D object recognition datasets and present a comparative performance evaluation of the proposed methods on these benchmark datasets. Finally, we identify promising avenues of research in this rapidly evolving field. This survey will not only enable researchers to get a good overview of the state-of-the-art methods for RGB-D-based object recognition but also provide a reference for other multimodal machine learning applications, e.g., multimodal medical image fusion, audiovisual speech recognition, and multimedia retrieval and generation.

show abstract

Section: ) Multimodal Fusion Strategiesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

RGB-D-Based Object Recognition Using Multimodal Convolutional Neural Networks: A Survey

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Sanchez et al 29 presented a comparative study of data fusion for RGB-D recognition. They compared the performances of different fusion techniques and different feature extraction approaches.…”

Section: Related Workmentioning

confidence: 99%

A PCA–CCA network for RGB-D object recognition

Sun

Zhao

et al. 2018

International Journal of Advanced Robotic Systems

View full text Add to dashboard Cite

Object recognition is one of the essential issues in computer vision and robotics. Recently, deep learning methods have achieved excellent performance in red-green-blue (RGB) object recognition. However, the introduction of depth information presents a new challenge: How can we exploit this RGB-D data to characterize an object more adequately? In this article, we propose a principal component analysis-canonical correlation analysis network for RGB-D object recognition. In this new method, two stages of cascaded filter layers are constructed and followed by binary hashing and block histograms. In the first layer, the network separately learns principal component analysis filters for RGB and depth. Then, in the second layer, canonical correlation analysis filters are learned jointly using the two modalities. In this way, the different characteristics of the RGB and depth modalities are considered by our network as well as the characteristics of the correlation between the two modalities. Experimental results on the most widely used RGB-D object data set show that the proposed method achieves an accuracy which is comparable to state-of-the-art methods. Moreover, our method has a simpler structure and is efficient even without graphics processing unit acceleration.

show abstract

“…There are studies that dedicated in researching a more efficient way to track an object. The study [17] compares the performance of the system that uses early or the late fusion with SVM (support vector machine) or other deep learning classifiers. In the case of hand gesture recognizing, the research [18] uses the deep learning to enhance the system to track the moving hand with faster speed without losing precision.…”

Section: Related Workmentioning

confidence: 99%

“…Hence, δ P can be found by solving Eq. (17) and the value of δ Q can be found by the bottom half equation of (16) while the value of δ P is already known.…”

Section: Bundle Adjustmentmentioning

confidence: 99%

A fast 3D scene reconstructing method using continuous video

Sung

Lin

2017

J Image Video Proc.

View full text Add to dashboard Cite

Accurate 3D measuring systems thrive in the past few years. Most of them are based on laser scanners because these laser scanners are able to acquire 3D information directly and precisely in real time. However, comparing to the conventional cameras, these kinds of equipment are usually expensive and they are not commonly available to customers. Moreover, laser scanners interfere easily with each other sensors of the same type. On the other hand, computer vision-based 3D measuring techniques use stereo matching to acquire the cameras' relative position and then estimate the 3D location of points on the image. Because this kind of systems needs additional estimation of the 3D information, systems with real time capability often relies on heavy parallelism that prevents implementation on mobile devices. Inspired by the structure from motion systems, we propose a system that reconstructs sparse feature points to a 3D point cloud using a mono video sequence so as to achieve higher computation efficiency. The system keeps tracking all detected feature points and calculates both the amount of these feature points and their moving distances. We only use the key frames to estimate the current position of the camera in order to reduce the computation load and the noise interference on the system. Furthermore, for the sake of avoiding duplicate 3D points, the system reconstructs the 2D point only when the point shifts out of the boundary of a camera. In our experiments, we show that our system is able to be implemented on tablets and can achieve state-of-the-art accuracy with a denser point cloud with high speed.

show abstract

A comparative study of data fusion for RGB-D based visual recognition

Cited by 59 publications

References 5 publications

RGB-D-Based Object Recognition Using Multimodal Convolutional Neural Networks: A Survey

RGB-D-Based Object Recognition Using Multimodal Convolutional Neural Networks: A Survey

A PCA–CCA network for RGB-D object recognition

A fast 3D scene reconstructing method using continuous video

Contact Info

Product

Resources

About