From passive to interactive object learning and recognition through self-identification on a humanoid robot

Lyubova, Natalia; Ivaldi, Serena; Filliat, David

doi:10.1007/s10514-015-9445-0

Cited by 15 publications

(7 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other set of works explore when the robot is able to interact also with the objects. Lyubova et al [19] learns objects models using point-feature descriptors and Bag of Words (BoW) models in two steps. The first one is based on just observation (either from the table or from a human showing the object) and the second one includes the robot interaction with the objects.…”

Section: A Robot Interactionmentioning

confidence: 99%

Incremental Learning of Object Models From Natural Human–Robot Interactions

Azagra

Civera

Murillo

2020

IEEE Trans. Automat. Sci. Eng.

View full text Add to dashboard Cite

In order to perform complex tasks in realistic human environments, robots need to be able to learn new concepts in the wild, incrementally, and through their interactions with humans. This paper presents an end-to-end pipeline to learn object models incrementally during the human-robot interaction.The pipeline we propose consists of three parts: (a) recognizing the interaction type, (b) detecting the object that the interaction is targeting, and (c) learning incrementally the models from data recorded by the robot sensors. Our main contributions lie in the target object detection, guided by the recognized interaction, and in the incremental object learning. The novelty of our approach is the focus on natural, heterogeneous and multimodal human-robot interactions to incrementally learn new object models. Throughout the paper we highlight the main challenges associated with this problem, such as high degree of occlusion and clutter, domain change, low resolution data and interaction ambiguity. Our work shows the benefits of using multi-view approaches and combining visual and language features, and our experimental results outperform standard baselines.Note to Practitioners-This work was motivated by challenges in recognition tasks for dynamic and varying scenarios. Our approach learns to recognize new user interactions and objects. To do so, we use multimodal data from the user-robot interaction: visual data is used to learn the objects and speech is used to learn the label and help with the interaction type recognition. We use state-of-the-art deep learning models to segment the user and the objects in the scene. Our algorithm for incremental learning is based on a classic incremental clustering approach.The pipeline we propose works with all sensors mounted on the robot, so it allows mobility on the system. Our work uses data recorded from a Baxter robot, which enables the use of the manipulation arms in future steps, but it would work with any robot able to have the same sensors mounted. The sensors used are two RGB-D cameras and a microphone. The pipeline currently has high computational requirements to run the two deep learning based steps. We have tested it with a desktop computer including a GTX 1060 and 32GB of RAM.

show abstract

Section: A Robot Interactionmentioning

confidence: 99%

Incremental Learning of Object Models From Natural Human–Robot Interactions

Azagra

Civera

Murillo

2020

IEEE Trans. Automat. Sci. Eng.

View full text Add to dashboard Cite

show abstract

“…The robot subjects these bit strings to a single analysis that returns 'yes' or 'no' for the presence of the target part. This analysis has multiple components, e.g., specification of a three-dimensional shape together with rotations and projections that accomplish "inverse optics" from the visual image (see e.g., [82] for an implementation of recognition of not one but several objects). These constitute the robot's reference measurements {M (R) i } Robot ; the pointer measurements then specify the position for grasping the part.…”

Section: What Information Is Collected?mentioning

confidence: 99%

“…She must, moreover, devote some of her observational overhead to observing herself in order to update her control system on where she is relative to whatever else she sees and how she is moving. The robot has no need for such self-observation (though again see [82] for a robot that must distinguish its own motions from those of another actor).…”

Section: What Information Is Collected?mentioning

confidence: 99%

Sciences of Observation

Fields

2018

Philosophies

View full text Add to dashboard Cite

Multiple sciences have converged, in the past two decades, on a hitherto mostly unremarked question: what is observation? Here, I examine this evolution, focusing on three sciences: physics, especially quantum information theory, developmental biology, especially its molecular and “evo-devo” branches, and cognitive science, especially perceptual psychology and robotics. I trace the history of this question to the late 19th century, and through the conceptual revolutions of the 20th century. I show how the increasing interdisciplinary focus on the process of extracting information from an environment provides an opportunity for conceptual unification, and sketch an outline of what such a unification might look like.

show abstract

“…A visuo-motor classifier is implemented in [4] in order to learn 5 different types of grasping gestures on 7 object types, by training an SVM model with object feature clusters (using K-means clustering) and a second SVM with 22 motor features (provided by a Cy-berGlove); the predictions are fused with a weighted linear combination of Mercer kernels. Moreover, in the field of robotics, affordance-related object recognition has relied on predicting opportunities for interaction with an object by using visual clues [1,11] or observing the effects of exploratory actions [20,23].…”

Section: Related Workmentioning

confidence: 99%

Deep Affordance-Grounded Sensorimotor Object Recognition

Thermos

Papadopoulos

Daras

et al. 2017

2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

It is well-established by cognitive neuroscience that human perception of objects constitutes a complex process, where object appearance information is combined with evidence about the so-called object "affordances", namely the types of actions that humans typically perform when interacting with them. This fact has recently motivated the "sensorimotor" approach to the challenging task of automatic object recognition, where both information sources are fused to improve robustness. In this work, the aforementioned paradigm is adopted, surpassing current limitations of sensorimotor object recognition research. Specifically, the deep learning paradigm is introduced to the problem for the first time, developing a number of novel neuro-biologically and neuro-physiologically inspired architectures that utilize state-of-the-art neural networks for fusing the available information sources in multiple ways. The proposed methods are evaluated using a large RGB-D corpus, which is specifically collected for the task of sensorimotor object recognition and is made publicly available. Experimental results demonstrate the utility of affordance information to object recognition, achieving an up to 29% relative error reduction by its inclusion.

show abstract

From passive to interactive object learning and recognition through self-identification on a humanoid robot

Cited by 15 publications

References 64 publications

Incremental Learning of Object Models From Natural Human–Robot Interactions

Incremental Learning of Object Models From Natural Human–Robot Interactions

Sciences of Observation

Deep Affordance-Grounded Sensorimotor Object Recognition

Contact Info

Product

Resources

About