Dual Online Stein Variational Inference for Control and Dynamics

Abstract

Model predictive control (MPC) schemes have a proven track record for delivering aggressive and robust performance in many challenging control tasks, coping with nonlinear system dynamics, constraints, and observational noise. Despite their success, these methods often rely on simple control distributions, which can limit their performance in highly uncertain and complex environments. MPC frameworks must be able to accommodate changing distributions over system parameters, based on the most recent measurements. In this paper, we devise an implicit variational inference algorithm able to estimate distributions over model parameters and control inputs on-the-fly. The method incorporates Stein Variational gradient descent to approximate the target distributions as a collection of particles, and performs updates based on a Bayesian formulation. This enables the approximation of complex multi-modal posterior distributions, typically occurring in challenging and realistic robot navigation tasks. We demonstrate our approach on both simulated and realworld experiments requiring real-time execution in the face of dynamically changing environments.

Robot Learning From Randomized Simulations: A Review

“…Continuing the direction of adaptive domain randomization, we are convinced that neural LFI powered by normalizing flows are auspicious approaches. The combination of highly flexible density estimators with widely applicable and sample-efficient inference methods allows one to identify multi-modal distributions over simulators with very mild assumptions ( Ramos et al, 2019 ; Barcelos et al, 2021 ; Muratore et al, 2021c ). By introducing an auxiliary optimality variable and making the policy parameters subject to the inference, we obtain the posterior over policies quantifying their likelihood of being optimal.…”

DiSProD: Differentiable Symbolic Propagation of Distributions for Planning

“…The same idea has been used before in MPPI Williams et al [2017], Wagener et al [2019]. Note that MPPI also uses multiple samples, but these samples all contribute to the update of a single action sequence, which can potentially limit the search , Barcelos et al [2021]. To allow reuse of old search but add diversity, we initialize one restart using the old actions shifted one time step, and initialize all other restarts randomly.…”

Variational Inference MPC using Normalizing Flows and Out-of-Distribution Projection

“…Two widely used sampling based MPC techniques, MPPI [36] and CEM [15], use importance sampling to generate low-cost control sequences, and have strong connections to the inference formulation of SOC which was explored in [34]. Several recent papers have considered the SOC problem as Bayesian inference, and proposed methods of performing Variational Inference (VI) to approximate a posterior over low-cost control sequences [17,34,23,3]. These methods differ in how they represent the variational posterior.…”

“…VI methods often use an independent Gaussian posterior, known as the mean-field approximation [4]. Okada and Taniguchi [23] represent the control sequence as a Gaussian mixture, and Lambert et al [17] use a particle representation, extended to handle parameter uncertainty in [3]. These representations allow for greater flexibility in representing complex posteriors.…”

“…Previous work has investigated the duality between control and inference [31,30] and considered both planning and control as inference problems [2,33,27]. Several recent papers have considered the finite-horizon stochastic optimal control problem as Bayesian inference, and proposed methods of performing variational inference to approximate the distribution used to sample control sequences [17,34,23,3]. In order to perform variational inference, we must specify a parameterized distribution which is tractable to optimize and sample while also [1].…”