Visual–Tactile Fusion for Object Recognition

Abstract

The camera provides rich visual information regarding objects and becomes one of the most mainstream sensors in the automation community. However, it is often difficult to be applicable when the objects are not visually distinguished. On the other hand, tactile sensors can be used to capture multiple object properties, such as textures, roughness, spatial features, compliance, and friction, and therefore provide another important modality for the perception. Nevertheless, effective combination of the visual and tactile modalities is still a challenging problem. In this paper, we develop a visualtactile fusion framework for object recognition tasks. This paper uses the multivariate-time-series model to represent the tactile sequence and the covariance descriptor to characterize the image. Further, we design a joint group kernel sparse coding (JGKSC) method to tackle the intrinsically weak pairing problem in visual-tactile data samples. Finally, we develop a visual-tactile data set, composed of 18 household objects for validation. The experimental results show that considering both visual and tactile inputs is beneficial and the proposed method indeed provides an effective strategy for fusion. Note to Practitioners-Visual and tactile measurements offer complementary properties that make them particularly suitable for fusion in order to address the robust and accurate recognition of objects, which is a necessity for many automation systems.In this paper, we investigate a widely applicable scenario in grasp manipulation. When identifying an object, the manipulator may see it using the camera and touch it using its hand. Thus, we obtain a pair of test samples, including one image sample and one tactile sample. The manipulator then utilizes this sample pair to identify this object with a classifier that is constructed using the previously collected training samples. However, when collecting training samples, we may collect the image samples and the tactile samples separately. In other words, the training samples may not be paired, while the test samples are paired. This paper addresses this practical problem by developing a JGKSC method, which encourages the effects of the same group, but different atoms. Although our focus is on combining visual and tactile information, the described problem framework is common in the automation community. The algorithm described in this paper can therefore work with weak pairings between a variety of sensors.

Precision Security: Integrating Video Surveillance with Surrounding Environment Changes

“…Objects understanding and detection in dynamic environment changes are usually based on the adaptive background subtraction and other objects recognition methods [17,21,35,[65][66][67][68]. A preliminary scheme for the practical integration of proposed models with these algorithms is presented in Figure 7, where smog as a global environmental change has significant implications to video behaviors recognition and loitering detection within a hovering period of two persons; only half of hovering behaviors is detected; only one person is red-highlighted and the other person is always in a green rectangle, indicating the degradation of video surveillance efficiency within the considered periods under any real challenging scenarios.…”

Robotic teleoperation systems using a wearable multimodal fusion device

“…They are more effective and more convenient to fulfill the tasks. In future, the tactile information of robotic hand can be used, [16][17][18] which will improve the grasp for teleoperation. …”

Natural multimodal communication for human–robot collaboration

“…Dobrišek et al 20 show how audio and video combination for emotion recognition increases the performance to 77.5%, increasing in 5% with respect to the best of the individual channels, that is, audio. Liu et al 21 use visual information in combination with information from tactile sensors to capture multiple object properties, such as textures, roughness, spatial features, compliance, and friction, bridging the gap for objects that are not visually distinguishable, leading to better recognition results without any relevant drawback in execution time.…”