Learning Human Objectives from Sequences of Physical Corrections

Abstract

When personal, assistive, and interactive robots make mistakes, humans naturally and intuitively correct those mistakes through physical interaction. In simple situations, one correction is sufficient to convey what the human wants. But when humans are working with multiple robots or the robot is performing an intricate task often the human must make several corrections to fix the robot's behavior. Prior research assumes each of these physical corrections are independent events, and learns from them one-at-a-time. However, this misses out on crucial information: each of these interactions are interconnected, and may only make sense if viewed together. Alternatively, other work reasons over the final trajectory produced by all of the human's corrections. But this method must wait until the end of the task to learn from corrections, as opposed to inferring from the corrections in an online fashion. In this paper we formalize an approach for learning from sequences of physical corrections during the current task. To do this we introduce an auxiliary reward that captures the human's trade-off between making corrections which improve the robot's immediate reward and long-term performance. We evaluate the resulting algorithm in remote and in-person human-robot experiments, and compare to both independent and final baselines. Our results indicate that users are best able to convey their objective when the robot reasons over their sequence of corrections.

Genetic Characteristics of Three Unique Recombinant Forms of HIV-1 in Yunnan, China

“…Notably, correctional feedback has been used in robotics applications (Li et al, 2021;Losey et al, 2021;Bajcsy et al, 2018;Ross et al, 2011). However, these robotics-focused works focus on improving a trajectory or a sequence of actions, which is a multi-step bandit problem.…”

“…Layout designs are used by engineers and designers to produce vectorized arrangements and models. There are several commonly used datasets for layout modeling, including Pub-layNet (Zhong et al, 2019), CLAY (Li et al, 2021), and RICO-Semantic (Sunkara et al, 2022). In this paper, we focus on UI layouts, which consist of a collection of UI elements.…”

Disentangling syntactics, semantics, and pragmatics in natural language processing

“…RLHF is a type of reinforcement learning that learns from human feedback, which encourage the alignment of their learning objectives with human values [185]. Existing feedback mechanisms are diverse to suit different learning objectives, including critiques [143,371,382], comparisons [88,254,362,374], improvements [153,187,196], natural language [2,199,337,383], etc. However, these approaches mostly focus on improving model performance, neglecting the explainability.…”