Aligning Robot and Human Representations

Abstract

To act in the world, robots rely on a representation of salient task aspects: for example, to carry a cup of coffee, a robot must consider movement efficiency and cup orientation in its behaviour. However, if we want robots to act for and with people, their representations must not be just functional but also reflective of what humans care about, i.e. their representations must be aligned with humans'. In this survey, we pose that current reward and imitation learning approaches suffer from representation misalignment, where the robot's learned representation does not capture the human's true representation. We suggest that because humans will be the ultimate evaluator of robot performance in the world, it is critical that we explicitly focus our efforts on aligning learned task representations with humans, in addition to learning the downstream task. We advocate that current representation learning approaches in robotics should be studied from the perspective of how well they accomplish the objective of representation alignment. To do so, we mathematically define the problem, identify its key desiderata, and situate current robot learning methods within this formalism. We conclude the survey by suggesting future directions for exploring open challenges.

Preference-Conditioned Language-Guided Abstraction

“…There are also works that actively learn from human teachers, where the emphasis is on generating actions or queries that are maximally informative for the human to label [6,12]. Unfortunately, these approaches all are limited by the fact that the feedback asked of the human is overfit to specific failures or desired data points, and rarely scale well relative to human time or effort [7].…”

FARPLS: A Feature-Augmented Robot Trajectory Preference Labeling System to Assist Human Labelers’ Preference Elicitation

“…Recent research has pointed out the potential misalignment issue between human values and robotic objectives. Studies to address this issue include bidirectional human-robot communication in group settings [83], evaluation of task accomplishment [6], and disentangled representation learning (DRL) [77]. Particularly, Reinforcement Learning from Human Preference (RLHP) [1,14,49,50,84] emerges as a new trend to offer a flexible and adaptable way to fine-tune an agent's behavior based on human preference.…”

“…However, these delicate reward functions may not accurately reflect humans' true values [18] due to generalization errors [57], task misspecifications [14], etc. The human-robot value alignment can not only improve robot performances according to humans' preference but also avoid undesired robot behavior and even safety issues [6,68,83]. Learning a reward model from human preferences hence emerges [18], which leverages the computationally efficient and user-friendly pairwise comparison to collect human preferences [8,14,38,39].…”