Robust semantic representations for inferring human co-manipulation activities even with different demonstration styles

Abstract

In this work we present a novel method that generates compact semantic models for inferring human coordinated activities, including tasks that require the understanding of dual arms sequencing. These models are robust and invariant to observation from different executions styles of the same activity. Additionally, the obtained semantic representations are able to re-use the acquired knowledge to infer different types of activities. Furthermore, our method is capable to infer dualarm co-manipulation activities and it considers the correct synchronization between the inferred activities to achieve the desired common goal. We propose a system that, rather than focusing on the different execution styles, extracts the meaning of the observed task by means of semantic representations. The proposed method is a hierarchical approach that first extracts the relevant information from the observations. Then, it infers the observed human activities based on the obtained semantic representations. After that, these inferred activities can be used to trigger motion primitives in a robot to execute the demonstrated task. In order to validate the portability of our system, we have evaluated our semantic-based method on two different humanoid platforms, the iCub robot and REEM-C robot. Demonstrating that our system is capable to correctly segment and infer on-line the observed activities with an average accuracy of 84.8%.

Tool-Use Model to Reproduce the Goal Situations Considering Relationship Among Tools, Objects, Actions and Effects Using Multimodal Deep Neural Networks

“…In other words, tooluse arises not only from tool features, but also from the operated objects, actions to take and the expected effects (i.e., goal). Acquisition of affordance is one of the most essential topic in the cognitive field Bushnell and Boudreau (1993), Piaget (1952), Rat-Fischer et al (2012), and Lockman (2000), and has also been discussed in robotics field and has gained significant support Horton et al (2012), Min et al (2016), and Jamone et al (2018). Inspired from this theory, we construct the tool-use model with DNNs simultaneously considering features of tools, objects, actions,and effects.…”

“…Following those models, the robots calculate and move along optimal motion trajectories. This approach has realized highly accurate and fast movements, allowing robots to perform pan tosses Pan et al (2018) , make pancakes with cooking tools Beetz et al (2011) , cut bread with a knife Ramirez-Amaro et al (2015) , and serve food with a spatula Nagahama et al (2013) in past studies. This approach can be efficient and realize high performance if the robots are dedicated to a particular purpose in a specific environment, but as the number of tools and objects or task types increase, it becomes difficult to design numerical models of properties and environments for each condition.…”

Robot Skin: Fully-Compliant Control Framework Using Multi-modal Tactile Events

“…These three fusion levels provide additional robustness on the generated data, especially to deal with partial information and the multi-sampling nature of a complex robotic system Ramirez-Amaro et al (2015b).…”

A Semi-Autonomous Tele-Impedance Method based on Vision and Voice Interfaces

“…Vision and visual feedback have been successfully used in robot learning, where the aim is to make robots autonomous. For example, the method [14] used visual feedback to enable the robot to learn and adjust complex behavioural semantics.…”