Deep Neural Object Analysis by Interactive Auditory Exploration with a Humanoid Robot

Eppe, Manfred; Kerzel, Matthias; Strahl, Erik; Wermter, Stefan

doi:10.1109/iros.2018.8593838

Cited by 17 publications

(8 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent techniques in visual question answering [49] and language grounding [55,32] allow robots to answer questions about objects and describe objects with natural language. Haptic [45,40] and auditory data [20,25] have also helped robots interpret salient features of objects beyond vision. Interactive perception can further leverage a robot's exploratory actions to reveal sensory signals that are otherwise not observable [9,56,14,61,2,8,59].…”

Section: A Semantic Reasoning In Roboticsmentioning

confidence: 99%

Learning Instance-Level N-Ary Semantic Knowledge At Scale For Robots Operating in Everyday Environments

Liu¹,

Bansal²,

Daruna³

et al. 2021

Robotics: Science and Systems XVII

View full text Add to dashboard Cite

Robots operating in everyday environments need to effectively perceive, model, and infer semantic properties of objects. Existing knowledge reasoning frameworks only model binary relations between an object's class label and its semantic properties, unable to collectively reason about object properties detected by different perception algorithms and grounded in diverse sensory modalities. We bridge the gap between multimodal perception and knowledge reasoning by introducing an n-ary representation that models complex, inter-related object properties. To tackle the problem of collecting n-ary semantic knowledge at scale, we propose a transformer neural network that directly generalizes knowledge from observations of object instances. The learned model can reason at different levels of abstraction, effectively predicting unknown properties of objects in different environmental contexts given different amounts of observed information. We quantitatively validate our approach against five prior methods on LINK, a unique dataset we contribute that contains 1457 situated object instances with 15 multimodal properties types and 200 total properties. Compared to the top-performing baseline, a Markov Logic Network, our model obtains a 10% improvement in predicting unknown properties of novel object instances while reducing training and inference time by 150 times. Additionally, we apply our work to a mobile manipulation robot, demonstrating its ability to leverage n-ary reasoning to retrieve objects and actively detect object properties. The code and data are available at https://github.com/wliu88/LINK.

show abstract

Section: A Semantic Reasoning In Roboticsmentioning

confidence: 99%

Learning Instance-Level N-Ary Semantic Knowledge At Scale For Robots Operating in Everyday Environments

Liu¹,

Bansal²,

Daruna³

et al. 2021

Robotics: Science and Systems XVII

View full text Add to dashboard Cite

show abstract

“…Vision-based recognition of an object is the commonly adopted approach; however, several research studies show incorporating a variety of sensory modalities is the key to further enhance the robotic capabilities in recognizing the multisensory object properties (see Bohg et al, 2017 ; Li et al, 2020 for a review). Previous work has shown that robots can recognize objects using non-visual sensory modalities such as the auditory (Torres-Jara et al, 2005 ; Sinapov et al, 2009 ; Luo et al, 2017 ; Eppe et al, 2018 ; Jin et al, 2019 ; Gandhi et al, 2020 ), the tactile (Sinapov et al, 2011b ; Bhattacharjee et al, 2012 ; Fishel and Loeb, 2012 ; Kerzel et al, 2019 ), and the haptic sensory modalities (Natale et al, 2004 ; Bergquist et al, 2009 ; Braud et al, 2020 ). In addition to recognizing objects, multisensory feedback has also proven useful for learning object categories (Sinapov et al, 2014a ; Högman et al, 2016 ; Taniguchi et al, 2018 ; Tatiya and Sinapov, 2019 ), material properties (Erickson et al, 2017 , 2019 ; Eguíluz et al, 2018 ), object relations (Sinapov et al, 2014b , 2016 ), and more generally, grounding linguistic descriptors (e.g., nouns and adjectives) that humans use to describe objects (Thomason et al, 2016 ; Richardson and Kuchenbecker, 2019 ; Arkin et al, 2020 ).…”

Section: Related Workmentioning

confidence: 99%

A Framework for Sensorimotor Cross-Perception and Cross-Behavior Knowledge Transfer for Object Categorization

et al. 2020

View full text Add to dashboard Cite

From an early age, humans learn to develop an intuition for the physical nature of the objects around them by using exploratory behaviors. Such exploration provides observations of how objects feel, sound, look, and move as a result of actions applied on them. Previous works in robotics have shown that robots can also use such behaviors (e.g., lifting, pressing, shaking) to infer object properties that camera input alone cannot detect. Such learned representations are specific to each individual robot and cannot currently be transferred directly to another robot with different sensors and actions. Moreover, sensor failure can cause a robot to lose a specific sensory modality which may prevent it from using perceptual models that require it as input. To address these limitations, we propose a framework for knowledge transfer across behaviors and sensory modalities such that: (1) knowledge can be transferred from one or more robots to another, and, (2) knowledge can be transferred from one or more sensory modalities to another. We propose two different models for transfer based on variational auto-encoders and encoder-decoder networks. The main hypothesis behind our approach is that if two or more robots share multi-sensory object observations of a shared set of objects, then those observations can be used to establish mappings between multiple features spaces, each corresponding to a combination of an exploratory behavior and a sensory modality. We evaluate our approach on a category recognition task using a dataset in which a robot used 9 behaviors, coupled with 4 sensory modalities, performed multiple times on 100 objects. The results indicate that sensorimotor knowledge about objects can be transferred both across behaviors and across sensory modalities, such that a new robot (or the same robot, but with a different set of sensors) can bootstrap its category recognition models without having to exhaustively explore the full set of objects.

show abstract

“…And it is proved that the sound of shaking can be used for object recognition in many places such as shopping malls, workshops and home. Eppe et al (2018) used a humanoid robot to perform auditory exploration of a group of visually indistinguishable plastic containers filled with different amounts of different materials, proving that deep recursive neural structures can learn to distinguish individual materials and estimate their weight.…”

Section: Related Workmentioning

confidence: 99%

“…The feature extraction method can also effectively reduce the dimension, thus reducing the computational cost. Related studies have successfully applied the Mel-Frequency Cepstral Coefficients (MFCCs) to speech feature extraction and object recognition, as in the literature (Nakamura et al, 2013; Luo et al, 2017; Eppe et al, 2018). The standard MFCC feature can only propose the static characteristics of the sound (Cao et al, 2017).…”

Section: Acoustic Dataset Collectionmentioning

confidence: 99%

Open-Environment Robotic Acoustic Perception for Object Recognition

et al. 2019

View full text Add to dashboard Cite

Object recognition in containers is extremely difficult for robots. Dynamic audio signals are more responsive to an object's internal property. Therefore, we adopt the dynamic contact method to collect acoustic signals in the container and recognize objects in containers. Traditional machine learning is to recognize objects in a closed environment, which is not in line with practical applications. In real life, exploring objects is dynamically changing, so it is necessary to develop methods that can recognize all classes of objects in an open environment. A framework for recognizing objects in containers using acoustic signals in an open environment is proposed, and then the kernel k nearest neighbor algorithm in an open environment (OSKKNN) is set. An acoustic dataset is collected, and the feasibility of the method is verified on the dataset, which greatly promotes the recognition of objects in an open environment. And it also proves that the use of acoustic to recognize objects in containers has good value.

show abstract

Deep Neural Object Analysis by Interactive Auditory Exploration with a Humanoid Robot

Cited by 17 publications

References 18 publications

Learning Instance-Level N-Ary Semantic Knowledge At Scale For Robots Operating in Everyday Environments

Learning Instance-Level N-Ary Semantic Knowledge At Scale For Robots Operating in Everyday Environments

A Framework for Sensorimotor Cross-Perception and Cross-Behavior Knowledge Transfer for Object Categorization

Open-Environment Robotic Acoustic Perception for Object Recognition

Contact Info

Product

Resources

About