Zeroth Order Non-convex optimization with Dueling-Choice Bandits

Abstract

We consider a novel setting of zeroth order non-convex optimization, where in addition to querying the function value at a given point, we can also duel two points and get the point with the larger function value. We refer to this setting as optimization with dueling-choice bandits since both direct queries and duels are available for optimization. We give the COMP-GP-UCB algorithm based on GP-UCB (Srinivas et al., 2009), where instead of directly querying the point with the maximum Upper Confidence Bound (UCB), we perform a constrained optimization and use comparisons to filter out suboptimal points. COMP-GP-UCB comes with theoretical guarantee of O( Φ √ T ) on simple regret where T is the number of direct queries and Φ is an improved information gain corresponding to a comparison based constraint set that restricts the search space for the optimum. In contrast, in the direct query only setting, Φ depends on the entire domain. Finally, we present experimental results to show the efficacy of our algorithm.

Preference-based Reinforcement Learning with Finite-Time Guarantees

“…PbRL is relevant to several settings in Multi-Armed Bandits. Dueling bandits [10,32] is essentially the one-state version of PbRL, and has been extensively studied in the literature [15,16,31,33]. However, PbRL is significantly harder because in PbRL the observation (preference) is based on the sum of rewards on a trajectory rather than individual reward values.…”