Bundled Gradients through Contact via Randomized Smoothing

Abstract

The empirical success of derivative-free methods in reinforcement learning for planning through contact seems at odds with the perceived fragility of classical gradient-based optimization methods in these domains. What is causing this gap, and how might we use the answer to improve gradient-based methods? We believe a stochastic formulation of dynamics is one crucial ingredient. We use tools from randomized smoothing to analyze sampling-based approximations of the gradient, and formalize such approximations through the gradient bundle. We show that using the gradient bundle in lieu of the gradient mitigates fast-changing gradients of non-smooth contact dynamics modeled by the implicit time-stepping, or the penalty method. Finally, we apply the gradient bundle to optimal control using iLQR, introducing a novel algorithm which improves convergence over using exact gradients. Combining our algorithm with a convex implicit time-stepping formulation of contact, we show that we can tractably tackle planningthrough-contact problems in manipulation.

Dojo: A Differentiable Simulator for Robotics

“…General-purpose LCP solvers that are typically used rely on a pivoting method like Lemke's algorithm [6]. Randomized smoothing has been proposed as a method for returning gradients through contact [32] with this model. An alternative soft-contact model is also available for patch contacts [9], but it is more computationally expensive, requiring sophisticated higher-order implicit integrators, and does not natively provide gradients.…”

“…MuJoCo overcomes such difficulties and returns continuous gradients by employing a soft-contact model and a finite difference scheme. For LCP-based simulators that return subgradients, randomized smoothing via gradient bundles has been proposed [32]. We compare Dojo's smooth gradients with the latter approach.…”