Learning rewards from exploratory demonstrations using probabilistic temporal ranking

Abstract

Informative path-planning is a well established approach to visual-servoing and active viewpoint selection in robotics, but typically assumes that a suitable cost function or goal state is known. This work considers the inverse problem, where the goal of the task is unknown, and a reward function needs to be inferred from exploratory example demonstrations provided by a demonstrator, for use in a downstream informative path-planning policy. Unfortunately, many existing reward inference strategies are unsuited to this class of problems, due to the exploratory nature of the demonstrations. In this paper, we propose an alternative approach to cope with the class of problems where these sub-optimal, exploratory demonstrations occur. We hypothesise that, in tasks which require discovery, successive states of any demonstration are progressively more likely to be associated with a higher reward, and use this hypothesis to generate time-based binary comparison outcomes and infer reward functions that support these ranks, under a probabilistic generative model. We formalise this probabilistic temporal ranking approach and show that it improves upon existing approaches to perform reward inference for autonomous ultrasound scanning, a novel application of learning from demonstration in medical imaging while also being of value across a broad range of goal-oriented learning from demonstration tasks.

Intelligent robotic sonographer: Mutual information-based disentangled reward learning from few demonstrations

“…Due to the acoustic shadows, poor contrast, speckle noise, and potential deformation in resulting images (Mishra et al, 2018), guiding a probe to the standard US planes is sophisticated, even for senior sonographers. This means the experts’ demonstrations of searching for standard US planes will be sub-optimal and even contradictory (Burke et al, 2023). Therefore, the popular maximum-entropy IRL method (Aghasadeghi and Bretl, 2011) cannot be directly applied in our applications.…”

“…In order to achieve good performance, it has strict requirements for the initial position and phantom position. Besides, Burke et al introduced a probabilistic temporal ranking model which assumes that the images shown in the later stage are more important than the earlier images (Burke et al, 2023), allowing for reward inference from sub-optimal scanning demonstrations. They use this model to coarsely navigate a US probe to a mimicked tumor inside of a gel phantom, followed by an exploratory Bayesian optimization policy to search for scanning positions that capture images with high rewards.…”

“…This is still a challenging task in the area of robot learning, particularly when the demonstrations are not optimal (Eteke et al, 2020). Inspired by Burke et al (2023), a global probabilistic spatial ranking (GPSR) method was developed to learn the latent skills and overcome the potentially biased results caused by inconsistent demonstrations. To increase the generalization capability and adapt the variation between patients, the MINE is employed (MI-GPSR) to explicitly disentangle the task-related features from the domain features.…”

“…The MI-GPSR mainly consists of two feature encoders to extract the task-related and domain features of the input images, a decoder to reconstruct the input images based on the extracted features in latent space, and a fully connected network (FCN) to predict the reward for individual images. Considering the time efficiency, and the safety concerns about the rich interactions with patients, it is impractical to train an agent using RL (Bi et al, 2022; Li et al, 2021a) or apply Gaussian process path planning (Burke et al, 2023) in a real scenario. Therefore, we divided the navigation process into coarse and fine-tuning procedures.…”

“…Probabilistic temporal ranking (PTR) allows for uncertainty quantification and reward inference from demonstrations (Burke et al, 2023). Considering the characteristic of US examinations, experts often need to move a probe around to search for the target anatomy and finally stop at some specific US planes to do biometric measurements.…”