Dueling RL: Reinforcement Learning with Trajectory Preferences

Abstract

We consider the problem of preference based reinforcement learning (PbRL), where, unlike traditional reinforcement learning, an agent receives feedback only in terms of a 1 bit (0/1) preference over a trajectory pair instead of absolute rewards for them. The success of the traditional RL framework crucially relies on the underlying agent-reward model, which, however, depends on how accurately a system designer can express an appropriate reward function and often a non-trivial task. The main novelty of our framework is the ability to learn from preferencebased trajectory feedback that eliminates the need to hand-craft numeric reward models. This paper sets up a formal framework for the PbRL problem with non-markovian rewards, where the trajectory preferences are encoded by a generalized linear model of dimension d. Assuming the transition model is known, we then propose an algorithm with almost optimal regret guarantee of Õ SHd log(T /δ) √ T . We further, extend the above algorithm to the case of unknown transition dynamics, and provide an algorithm with near optimal regret guarantee O((To the best of our knowledge, our work is one of the first to give tight regret guarantees for preference based RL problem with trajectory preferences.

Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation

“…Then the α-Eluder dimension of F T is at most O(dr 2 log(rLS h/α)). Therefore, our results subsume the setting of logistic preference functions (Pacchiano et al, 2021) as a spacial case.…”

“…The function spaces are general sets of functions, which may be either finitely parameterized or nonparametric. This setting is more general than the previous theoretical results for PbRL (Novoseller et al, 2020;Xu et al, 2020b;Pacchiano et al, 2021). Our contributions are summarized as follows:…”

“…Trajectory preference is the most general form of preference-based feedback, which is also the main focus of this work. As discussed in the introduction, previous theoretical results studying PbRL with trajectory feedback mainly focus on the tabular RL setting with finite state and action space (Novoseller et al, 2020;Xu et al, 2020b;Pacchiano et al, 2021). Besides PbRL, preferencebased learning has also been well-explored in bandit setting under the notion of "dueling bandits" (Yue et al, 2012;Falahatgar et al, 2017a;Busa-Fekete et al, 2018;Xu et al, 2020a;Busa-Fekete et al, 2018), which can be regarded as a special case of PbRL with single state and horizon H = 1.…”

“…Even worse, since we only get feedback about the preference between trajectory pairs, we cannot directly evaluate a single policy. Inspired from a recent work for PbRL in the tabular setting (Pacchiano et al, 2021), we construct a near-optimal policy set using the preference information.…”

“…Xu et al (2020b) presents the first finitetime analysis for PbRL problems with near-optimal sample complexity bounds. Pacchiano et al (2021) studies the regret minimization problem for PbRL with linearlyparameterized preference function. However, all the previous algorithms are restricted to the tabular setting, and their complexity bounds scale polynomial dependence on the cardinality of the state-action space.…”