Efficient Model Learning from Joint-Action Demonstrations for Human-Robot Collaborative Tasks

Abstract

We present a framework for automatically learning human user models from joint-action demonstrations that enables a robot to compute a robust policy for a collaborative task with a human. First, the demonstrated action sequences are clustered into different human types using an unsupervised learning algorithm. A reward function is then learned for each type through the employment of an inverse reinforcement learning algorithm. The learned model is then incorporated into a mixed-observability Markov decision process (MOMDP) formulation, wherein the human type is a partially observable variable. With this framework, we can infer online the human type of a new user that was not included in the training set, and can compute a policy for the robot that will be aligned to the preference of this user. In a human subject experiment (n = 30), participants agreed more strongly that the robot anticipated their actions when working with a robot incorporating the proposed framework (p < 0.01), compared to manually annotating robot actions. In trials where participants faced difficulty annotating the robot actions to complete the task, the proposed framework significantly improved team efficiency (p < 0.01). The robot incorporating the framework was also found to be more responsive to human actions compared to policies computed using a hand-coded reward function by a domain expert (p < 0.01). These results indicate that learning human user models from joint-action demonstrations and encoding them in a MOMDP formalism can support effective teaming in human-robot collaborative tasks.

Heterogeneous Learning from Demonstration

“…First, we consider a regular neural network f N N , which is our baseline. Second, we include the clustering-based method, denoted f N N (i) , by Nikolaidis et al [3] in which we cluster the gameplay data into three partitions using k-means and learn a separate policy network on each. Third, we consider a BNN, f BN N , which is able to holistically reason about the homo-and heterogeneity amongst the demonstrators.…”

“…[5] showed that when attempting LfD from pilots' demonstrations executing a single flight plan, averaging trajectories led to worse performance than using a single trajectory. Nikolaidis et al [3] approached this issue by categorizing demonstrators according to their task execution preference by clustering and learning a separate policy for each cluster. While this allows for utilization of the entire dataset, each policy learns off a fraction of the data.…”

An ensemble inverse optimal control approach for robotic task learning and adaptation

“…Here the intention is also assumed to be subject to a latent dynamics. In the work of Nikolaidis et al (2015), the robot learned to cooperate a painting task by holding and adjusting the pose of a cube according to the preferred sequence of human collaborators. Similar to our work, it employed inverse reinforcement learning over classified demonstrations with different styles.…”

“…Similar to our work, it employed inverse reinforcement learning over classified demonstrations with different styles. Our approach is distinct by using the learned ensemble as the mode observational model, which was user-specific in Nikolaidis et al (2015). Moreover, our system directly learns with continuous demonstration data, while Nikolaidis et al (2015) resorted to state discretization and the adopted mixed-observation Markov Decision Process is limited to low dimensional tasks.…”

Learning models of human-robot interaction from small data

“…Optimal policies for robots are then computed by forming a belief over human intention. In slightly different instances of applications of MOMDPs in HRI [22], the class of human subjects is considered as the unobservable variable. While it is true that in general MOMDPs drastically decrease the complexity of the problem, they usually require fully known submodels for each one of the unobservable variables.…”