Transferring skills to humanoid robots by extracting semantic representations from observations of human activities

Ramirez-Amaro, Karinne; Beetz, Michael; Cheng, Gordon

doi:10.1016/j.artint.2015.08.009

Cited by 108 publications

(59 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The average accuracy for the on-line segmentation and recognition of the overall activities for the scenarios shown in Table I is 84.8% for both hands. In our previous work [23], we concluded that the correct segmentation and recognition of human activities is not unique and greatly depends on the person interpreting the motions, especially when both hands are involved. The following link presents a video with more details about all these experimental results: http://web.ics.ei.tum.de/∼karinne/Videos/RamirezDChumanoids15.avi VIII.…”

Section: B Robot Execution Resultsmentioning

confidence: 99%

Robust semantic representations for inferring human co-manipulation activities even with different demonstration styles

Ramirez-Amaro

Dean-León

Cheng

2015

2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids)

Self Cite

View full text Add to dashboard Cite

In this work we present a novel method that generates compact semantic models for inferring human coordinated activities, including tasks that require the understanding of dual arms sequencing. These models are robust and invariant to observation from different executions styles of the same activity. Additionally, the obtained semantic representations are able to re-use the acquired knowledge to infer different types of activities. Furthermore, our method is capable to infer dualarm co-manipulation activities and it considers the correct synchronization between the inferred activities to achieve the desired common goal. We propose a system that, rather than focusing on the different execution styles, extracts the meaning of the observed task by means of semantic representations. The proposed method is a hierarchical approach that first extracts the relevant information from the observations. Then, it infers the observed human activities based on the obtained semantic representations. After that, these inferred activities can be used to trigger motion primitives in a robot to execute the demonstrated task. In order to validate the portability of our system, we have evaluated our semantic-based method on two different humanoid platforms, the iCub robot and REEM-C robot. Demonstrating that our system is capable to correctly segment and infer on-line the observed activities with an average accuracy of 84.8%.

show abstract

Section: B Robot Execution Resultsmentioning

confidence: 99%

Robust semantic representations for inferring human co-manipulation activities even with different demonstration styles

Ramirez-Amaro

Dean-León

Cheng

2015

2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The ability to perform actions based on observations of human activities is one of the major challenges to increase the capabilities of robotic systems [1]. Over the past few years, this problem has been of great interest to researchers and remains an active field in robotics [2]. By understanding human actions, robots may be able to acquire new skills, or perform different tasks, without the need for tedious programming.…”

Section: Introductionmentioning

confidence: 99%

Translating Videos to Commands for Robotic Manipulation with Deep Recurrent Neural Networks

Nguyen

Kanoulas

Muratore

et al. 2018

2018 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

We present a new method to translate videos to commands for robotic manipulation using Deep Recurrent Neural Networks (RNN). Our framework first extracts deep features from the input video frames with a deep Convolutional Neural Networks (CNN). Two RNN layers with an encoderdecoder architecture are then used to encode the visual features and sequentially generate the output words as the command. We demonstrate that the translation accuracy can be improved by allowing a smooth transaction between two RNN layers and using the state-of-the-art feature extractor. The experimental results on our new challenging dataset show that our approach outperforms recent methods by a fair margin. Furthermore, we combine the proposed translation module with the vision and planning system to let a robot perform various manipulation tasks. Finally, we demonstrate the effectiveness of our framework on a full-size humanoid robot WALK-MAN.

show abstract

“…Another important aspect of our system is its scalability and adaptability toward different domains as presented in [5], [32]. It is important to highlight that the semantic rules are obtained off-line and can be re-used directly in multiple domains as an on-line component.…”

Section: B Activity Inference From Robot Demonstrationsmentioning

confidence: 99%

“…In addition, we introduce a multi-modal control approach to enable different dynamic behaviors for standard industrial robots. The integration of these components is done with our novel semantic reasoning framework [5] to teach kinesthetically new activities to robots, see Fig. 1.…”

Section: Introductionmentioning

confidence: 99%

Robotic technologies for fast deployment of industrial robot systems

Dean-León

Ramirez-Amaro

Bergner

et al. 2016

IECON 2016 - 42nd Annual Conference of the IEEE Industrial Electronics Society

Self Cite

View full text Add to dashboard Cite

Abstract-The development of breakthrough technologies helps the deployment of robotic systems in the industry. The implementation and integration of such technologies will improve productivity, flexibility and competitiveness, in diverse industrial settings specially for small and medium enterprises. In this paper we present a framework that integrates three novel technologies, namely safe robot arms with multi-modal and auto-calibrated sensing skin, a robot control framework to generate dynamic behaviors fusing multiple sensor signals, and an intuitive and fast teaching by demonstration method that segments and recognizes the robot activities on-line based on re-usable semantic descriptions. In order to validate our framework, these technologies are integrated in a industrial setting to sort and pack fruits. We demonstrate that our presented framework enables a standard industrial robotic system to be flexible, modular and adaptable to different production requirements.

show abstract

Transferring skills to humanoid robots by extracting semantic representations from observations of human activities

Cited by 108 publications

References 17 publications

Robust semantic representations for inferring human co-manipulation activities even with different demonstration styles

Robust semantic representations for inferring human co-manipulation activities even with different demonstration styles

Translating Videos to Commands for Robotic Manipulation with Deep Recurrent Neural Networks

Robotic technologies for fast deployment of industrial robot systems

Contact Info

Product

Resources

About