On Entropy Regularized Path Integral Control for Trajectory Optimization

Abstract

In this article, we present a generalized view on Path Integral Control (PIC) methods. PIC refers to a particular class of policy search methods that are closely tied to the setting of Linearly Solvable Optimal Control (LSOC), a restricted subclass of nonlinear Stochastic Optimal Control (SOC) problems. This class is unique in the sense that it can be solved explicitly yielding a formal optimal state trajectory distribution. In this contribution, we first review the PIC theory and discuss related algorithms tailored to policy search in general. We are able to identify a generic design strategy that relies on the existence of an optimal state trajectory distribution and finds a parametric policy by minimizing the cross-entropy between the optimal and a state trajectory distribution parametrized by a parametric stochastic policy. Inspired by this observation, we then aim to formulate a SOC problem that shares traits with the LSOC setting yet that covers a less restrictive class of problem formulations. We refer to this SOC problem as Entropy Regularized Trajectory Optimization. The problem is closely related to the Entropy Regularized Stochastic Optimal Control setting which is often addressed lately by the Reinforcement Learning (RL) community. We analyze the theoretical convergence behavior of the theoretical state trajectory distribution sequence and draw connections with stochastic search methods tailored to classic optimization problems. Finally we derive explicit updates and compare the implied Entropy Regularized PIC with earlier work in the context of both PIC and RL for derivative-free trajectory optimization.

Entropy Regularised Deterministic Optimal Control: From Path Integral Solution to Sample-Based Trajectory Optimisation

“…Regardless, entropy regularisation turns out to be a fruitful direction for our purpose. Recent research illustrates how the principle of entropic inference can be put forth as a principled motivation for entropy regularisation in deterministic optimisation [24], [25]. In this paper we make a direct connection between entropy regularised optimisation and deterministic optimal control.…”

“…When we approximate beliefs π g using the Normal distribution N (x|µ, Σ); this approach generates the updates shown in algorithm 2. We refer to [24] for further details.…”

“…2) Why do heuristic adjustments of MPPI, not justified by theory, work better than the original algorithm? The first step towards answering these questions was taken in [24], where the underlying problem statement was still closely related to LSOC. As a result, the dynamics had to be invertible and the optimisation variable was a state transition distribution rather than a policy.…”

Trajectory Planning of Robot Manipulator Based on RBF Neural Network

“…With the advancements in automation and robot technology, robots have begun to be widely used in the industrial, agricultural, and medical fields, among many others. Improving the trajectory planning of robot manipulators is one of the core focuses of robot research, and has great research prospects [ 1 ]. Precise robot manipulator trajectories can improve the efficiency of a robot’s various tasks, such as workshop operations, crop picking, medical surgery and so on.…”

Exploratory LQG Mean Field Games with Entropy Regularization

“…Recently entropy regularized stochastic control problems [22,37,40,46] and stochastic games [42], where an entropy term is included in the objective function and the space of actions is the space of measures, have received attention in the literature. This research direction aims to provide a theoretical basis for the policy search techniques used in the field of reinforcement learning (RL), where an agent learns about an unknown environment by trial and error using feedback from their own actions and experiences.…”