Failure is Not an Option: Policy Learning for Adaptive Recovery in Space Operations

Abstract

This paper considers the problem of how robots in long-term space operations can learn to choose appropriate sources of assistance to recover from failures. Current assistant selection methods for failure handling are based on manually specified static look up tables or policies, which are not responsive to dynamic environments or uncertainty in human performance. We describe a novel and highly flexible learningbased assistant selection framework that uses contextual multiarm bandit algorithms. The contextual bandits exploit information from observed environment and assistant performance variables to efficiently learn selection policies under a wide set of uncertain operating conditions and unknown/dynamically constrained assistant capabilities. Proof of concept simulations of long-term human-robot interactions for space exploration are used to compare the performance of the contextual bandit against other state of the art assistant selection approaches. The contextual bandit outperforms conventional static policies and non-contextual learning approaches, and also demonstrates favorable robustness and scaling properties.

Everybody Needs Somebody Sometimes: Validation of Adaptive Recovery in Robotic Space Operations

“…Two matched sets of simulations were conducted. In each set, fully disjoint teams of actors were compared against shared teams selected from a set of ten actors, producing regret curves similar to our previously published work [1]. The number of selection events needed before the multi-arm bandit outperformed the informed static policy was then measured for one hundred trials in each setup.…”

“…b) Apparatus and Environment: A key challenge for this experimental design was ensuring that subjects completed a sufficient number of subtasks, so that our assistant selection algorithm had enough data to learn from. Based on our prior simulations [1], we found that the number of selection events required to achieve parity with a state-of-the-art approach scales linearly in both the size of the assistant population and the number of types of subtask. Therefore, the human data collection involved a small number of subtasks and used small assistant populations, to validate performance of our adaptive approach within the horizon imposed by the number of selection events possible in a single testing session.…”

“…In previous work [1], we used idealized simulations to show that indicators of opportunity can improve the performance of an assistant selection algorithm. Our intent in developing a human-sourced dataset is to capture the difficult-to-model properties of human task performance.…”

“…The estimates of β and θ a are then updated with that actor's observed reward. This policy was chosen for its simplicity and expressive power and was validated in our prior work [1]. In the informed static policy is determined from a post-hoc analysis of the ground truth behaviour data.…”

“…In [1], we validated the use of IOOs in an idealized simulation of an autonomous robot working together with human assistants in a space environments, using a contextaware reinforcement-learning (RL)-based policy for adaptive assistant selection. Using cumulative regret (the running sum of differences between rewards earned by a policy's choice of assistant and the highest possible rewards that could have been earned) as a measurement of performance, we demonstrated that the IOOs can be exploited with "black box" models of human performance to provide improved performance, as compared to existing baseline assistant selection policies that do not consider such contextual information.…”