POMDP Model Learning for Human Robot Collaboration

Abstract

Recent years have seen human robot collaboration (HRC) quickly emerged as a hot research area at the intersection of control, robotics, and psychology. While most of the existing work in HRC focused on either low-level human-aware motion planning or HRC interface design, we are particularly interested in a formal design of HRC with respect to high-level complex missions, where it is of critical importance to obtain an accurate and meanwhile tractable human model. Instead of assuming the human model is given, we ask whether it is reasonable to learn human models from observed perception data, such as the gesture, eye movements, head motions of the human in concern. As our initial step, we adopt a partially observable Markov decision process (POMDP) model in this work as mounting evidences have suggested Markovian properties of human behaviors from psychology studies. In addition, POMDP provides a general modeling framework for sequential decision making where states are hidden and actions have stochastic outcomes. Distinct from the majority of POMDP model learning literature, we do not assume that the state, the transition structure or the bound of the number of states in POMDP model is given. Instead, we use a Bayesian non-parametric learning approach to decide the potential human states from data. Then we adopt an approach inspired by probably approximately correct (PAC) learning to obtain not only an estimation of the transition probability but also a confidence interval associated to the estimation. Then, the performance of applying the control policy derived from the estimated model is guaranteed to be sufficiently close to the true model. Finally, data collected from a driver-assistance test-bed are used to train the model, which illustrates the effectiveness of the proposed learning method.

Getting to Know One Another: Calibrating Intent, Capabilities and Trust for Human-Robot Collaboration

“…In our scenario, the intention of the human (represented by θ) is unknown to the robot. Similar to other POMDP-based human-robot collaboration models [34], [35], a straightforward way to enable the robot to learn θ is to embed it into the state-we augment the POMDP's state with θ. Although θ is latent, human behavior is informative about its value; the human is assumed to act consistently to achieve her objectives.…”

“…For example, recent work [1], [30] has developed trust-based POMDPs that enable robots to consider the human's underlying trust in the robot during planning. Other works have contributed technical innovations for dealing with uncertainty and the complexity of HRC POMDPs, e.g., a Bayesian method to learn the state space and estimate the observation/transition functions [34], and hierarchical structure to reduce planning complexity [40]. Similar to our solution technique, [35] modified the vanilla POMCP for large observation spaces associated with HRC POMDPs.…”

Vector Autoregressive POMDP Model Learning and Planning for Human–Robot Collaboration

“…Compared with the HMM assumption which ignores the dependence among observations [10], the AR-HMM assumption makes the model be able to extract dynamic properties of the observing data since it considers the observing data as a dynamic system. Thus using the AR-HMM as the generative model is necessary when one cares about dynamic properties rather than static properties of the observing data.…”

“…In this case, modeling uncertainties will make the learned transition probabilities subject to a certain confidence level which motivates us to apply the Chernoff bound to reason the accuracy of the transition probabilities for VAR-POMDP. Details of the transitions probability learning can be found in [10]. From the Chernoff bound, the estimation error of the transition probability can be sufficiently small with high confidence as long as the training data is sufficient enough.…”

“…However, the state space could be tedious to be predefined and the number of hidden states could be case dependent especially when human is involved. In our previous work [10], we dropped these assumptions and proposed a Bayesian non-parametric learning approach to infer the structure of the POMDP model, such as the number of states, from data. However, hidden states of the POMDP model being learned can only model static properties of the observed data.…”