A Transfer Learning Approach to Cross-Modal Object Recognition: From Visual Observation to Robotic Haptic Exploration

Falco, Pietro; Lu, Shuang; Natale, Ciro; Pirozzi, Salvatore; Lee, Dongheui

doi:10.1109/tro.2019.2914772

Cited by 37 publications

(38 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Takahashi and Tan [128] and Gandarias et al [129] recently studied how to obtain tactile features from optical images based on self-encoding networks and CNNs and how to use them to enhance the tactile perception capability. Falco et al [130] established an active exploration framework to realize cross-modal visual-tactile object recognition. These results verified the possibility of using visual information to enhance tactile perception.…”

Section: Embodied Tactile Learningmentioning

confidence: 99%

Embodied tactile perception and learning

Liu

Guo

Sun

et al. 2020

Brain Science Advances

View full text Add to dashboard Cite

Various living creatures exhibit embodiment intelligence, which is reflected by a collaborative interaction of the brain, body, and environment. The actual behavior of embodiment intelligence is generated by a continuous and dynamic interaction between a subject and the environment through information perception and physical manipulation. The physical interaction between a robot and the environment is the basis for realizing embodied perception and learning. Tactile information plays a critical role in this physical interaction process. It can be used to ensure safety, stability, and compliance, and can provide unique information that is difficult to capture using other perception modalities. However, due to the limitations of existing sensors and perception and learning methods, the development of robotic tactile research lags significantly behind other sensing modalities, such as vision and hearing, thereby seriously restricting the development of robotic embodiment intelligence. This paper presents the current challenges related to robotic tactile embodiment intelligence and reviews the theory and methods of robotic embodied tactile intelligence. Tactile perception and learning methods for embodiment intelligence can be designed based on the development of new large‐scale tactile array sensing devices, with the aim to make breakthroughs in the neuromorphic computing technology of tactile intelligence.

show abstract

Section: Embodied Tactile Learningmentioning

confidence: 99%

Embodied tactile perception and learning

Liu

Guo

Sun

et al. 2020

Brain Science Advances

View full text Add to dashboard Cite

show abstract

“…Integrating visual and tactile information for object shape estimation [13]- [16] as well as object recognition [28] has been studied. However, these studies commonly assumed that the visual and tactile shapes are very similar to the actual object shape.…”

Section: Related Workmentioning

confidence: 99%

Exploiting Visual-Outer Shape for Tactile-Inner Shape Estimation of Objects Covered with Soft Materials

Miyamoto

Sasaki

Matsubara

2020

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

In this letter, we consider the problem of innershape estimation of objects covered with soft materials, e.g., pastries wrapped in paper or vinyl, water bottles covered with shock-absorbing fabrics, or human bodies dressed in clothes. Due to the softness of the covered materials, tactile information obtained through physical touches can be useful to estimate such inner shape; however, using only tactile information is inefficient since it can collect local information at around the touchpoint. Another approach would be taking visual information obtained by cameras into account; however, it is not straightforward since the visual information only captures the outer shape of the covered materials, and it is unknown how much such visualouter shape is similar/dissimilar to the tactile-inner shape. We propose an active tactile exploration framework that can utilize the visual-outer shape to efficiently estimate the inner shape of objects covered with soft materials. To this end, we propose the Gaussian Process Inner-Outer Implicit Surface model (GPIOIS) that jointly models the implicit surfaces of inner-outer shapes with their similarity by Gaussian processes. Simulation and realrobot experimental results demonstrated the effectiveness of our method.

show abstract

“…Especially, convolutional neural networks have become one of the most successful image-based pattern recognition methods [ 27 , 28 , 29 , 30 ]. A transfer learning approach is among most useful techniques for adapting pre-trained CNN architectures to other image domains [ 31 , 32 , 33 , 34 ]. With the aid of transfer learning, it is possible to train an effective deep neural network (DNN) architecture with a limited number of training samples because it is possible to reuse previously trained kernels.…”

Section: Introductionmentioning

confidence: 99%

Recognition of Cosmic Ray Images Obtained from CMOS Sensors Used in Mobile Phones by Approximation of Uncertain Class Assignment with Deep Convolutional Neural Network

Hachaj

Bibrzycki

Piekarczyk

2021

Sensors

View full text Add to dashboard Cite

In this paper, we describe the convolutional neural network (CNN)-based approach to the problems of categorization and artefact reduction of cosmic ray images obtained from CMOS sensors used in mobile phones. As artefacts, we understand all images that cannot be attributed to particles’ passage through sensor but rather result from the deficiencies of the registration procedure. The proposed deep neural network is composed of a pretrained CNN and neural-network-based approximator, which models the uncertainty of image class assignment. The network was trained using a transfer learning approach with a mean squared error loss function. We evaluated our approach on a data set containing 2350 images labelled by five judges. The most accurate results were obtained using the VGG16 CNN architecture; the recognition rate (RR) was 85.79% ± 2.24% with a mean squared error (MSE) of 0.03 ± 0.00. After applying the proposed threshold scheme to eliminate less probable class assignments, we obtained a RR of 96.95% ± 1.38% for a threshold of 0.9, which left about 62.60% ± 2.88% of the overall data. Importantly, the research and results presented in this paper are part of the pioneering field of the application of citizen science in the recognition of cosmic rays and, to the best of our knowledge, this analysis is performed on the largest freely available cosmic ray hit dataset.

show abstract

A Transfer Learning Approach to Cross-Modal Object Recognition: From Visual Observation to Robotic Haptic Exploration

Cited by 37 publications

References 41 publications

Embodied tactile perception and learning

Embodied tactile perception and learning

Exploiting Visual-Outer Shape for Tactile-Inner Shape Estimation of Objects Covered with Soft Materials

Recognition of Cosmic Ray Images Obtained from CMOS Sensors Used in Mobile Phones by Approximation of Uncertain Class Assignment with Deep Convolutional Neural Network

Contact Info

Product

Resources

About