Do Differentiable Simulators Give Better Policy Gradients?

Terry, Suh, H. J.; Simchowitz, Max; Zhang, Kaiqing; Tedrake, Russ

doi:10.48550/arxiv.2202.00817

Cited by 3 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Remark 5.5. Model-based RP PGMs with non-smooth models and policies can suffer from large variance and highly non-smooth loss landscapes, which can lead to slow convergence or failure during training even in simple toy examples (Parmas et al, 2018;Metz et al, 2021;Suh et al, 2022a). Proposition 5.4 suggests that one can add smoothness regularization to avoid exploding gradient variance.…”

Section: Resultsmentioning

confidence: 99%

“…Two main categories have emerged in the realm of stochastic gradient estimation: (1) Likelihood Ratio (LR) estimators, which perform zeroth-order estimation through the sampling of function evaluations (Williams, 1992;Konda and Tsitsiklis, 1999;Kakade, 2001), and (2) ReParameterization (RP) gradient estimators, which harness the differentiability of the function approximation (Figurnov et al, 2018;Ruiz et al, 2016;Clavera et al, 2020;Suh et al, 2022a). Despite the wide adoption of both LR and RP PGMs in practice, the majority of the literature on the theoretical properties of PGMs focuses on LR PGMs.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

New subspace minimization conjugate gradient methods based on regularization model for unconstrained optimization

Zhao

Liu

2020

Numer Algor

View full text Add to dashboard Cite

ReParameterization (RP) Policy Gradient Methods (PGMs) have been widely adopted for continuous control tasks in robotics and computer graphics. However, recent studies have revealed that, when applied to long-term reinforcement learning problems, model-based RP PGMs may experience chaotic and non-smooth optimization landscapes with exploding gradient variance, which leads to slow convergence. This is in contrast to the conventional belief that reparameterization methods have low gradient estimation variance in problems such as training deep generative models. To comprehend this phenomenon, we conduct a theoretical examination of model-based RP PGMs and search for solutions to the optimization difficulties. Specifically, we analyze the convergence of the model-based RP PGMs and pinpoint the smoothness of function approximators as a major factor that affects the quality of gradient estimation. Based on our analysis, we propose a spectral normalization method to mitigate the exploding variance issue caused by long model unrolls. Our experimental results demonstrate that proper normalization significantly reduces the gradient variance of model-based RP PGMs. As a result, the performance of the proposed method is comparable or superior to other gradient estimators, such as the Likelihood Ratio (LR) gradient estimator. Our code is available at https://github.com/agentification/RP_PGM.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

New subspace minimization conjugate gradient methods based on regularization model for unconstrained optimization

Zhao

Liu

2020

Numer Algor

View full text Add to dashboard Cite

show abstract

“…Our main goal is to study scenarios with deformables, because prior works already explored a number of tasks limited to rigid-body motion (see previous section). That said, very few works considered contact-rich tasks, and one prior work highlighted the potential fundamental limitations of gradients for rigid contacts [6]. Hence, there is a need to further study the fundamentals of how loss landscapes and gradient fields are affected by rigid contacts.…”

Section: Bo-leap: a Methods For Global Search On Rugged Landscapesmentioning

confidence: 99%

“…Surprisingly, efforts to investigate the differentiability of these simulators are far and few. One prior work [6] has highlighted a few fundamental limitations of differentiable simulation in the presence of rigid contacts, in low-dimensional systems. In this work, we first investigate the quality of gradients by visualizing loss landscapes through differentiable simulators for several robotic manipulation tasks of interest.…”

Section: Introductionmentioning

confidence: 99%

Rethinking Optimization with Differentiable Simulation from a Global Perspective

Antonova¹,

Yang²,

Jatavallabhula³

et al. 2022

Preprint

View full text Add to dashboard Cite

Differentiable simulation is a promising toolkit for fast gradient-based policy optimization and system identification. However, existing approaches to differentiable simulation have largely tackled scenarios where obtaining smooth gradients has been relatively easy, such as systems with mostly smooth dynamics. In this work, we study the challenges that differentiable simulation presents when it is not feasible to expect that a single descent reaches a global optimum, which is often a problem in contact-rich scenarios. We analyze the optimization landscapes of diverse scenarios that contain both rigid bodies and deformable objects. In dynamic environments with highly deformable objects and fluids, differentiable simulators produce rugged landscapes with nonetheless useful gradients in some parts of the space. We propose a method that combines Bayesian optimization with semi-local 'leaps' to obtain a global search method that can use gradients effectively, while also maintaining robust performance in regions with noisy gradients. We show that our approach outperforms several gradient-based and gradient-free baselines on an extensive set of experiments in simulation, and also validate the method using experiments with a real robot and deformables. Videos and supplementary materials are available at https://tinyurl.com/globdiff.

show abstract

“…Such an approach is mostly useful for perception-based control systems with a complex mapping h. It is also possible to apply autodifferentiation to obtain the gradient estimate. In general, it is unclear whether the finite-difference technique or the autodifferentiation method gives a better gradient in the context of control, and this topic is still being studied actively [27]. We will investigate this issue in the future.…”

Section: Model-free Roa Analysismentioning

confidence: 99%

Revisiting PGD Attacks for Stability Analysis of High-Dimensional Nonlinear Systems and Perception-Based Control

Havens

Kevian

Seiler

et al. 2023

IEEE Control Syst. Lett.

View full text Add to dashboard Cite

Many existing region-of-attraction (ROA) analysis tools find difficulty in addressing feedback systems with large-scale neural network (NN) policies and/or highdimensional sensing modalities such as cameras. In this letter, we tailor the projected gradient descent (PGD) attack method as a general-purpose ROA analysis tool for highdimensional nonlinear systems and end-to-end perceptionbased control. We show that the ROA analysis can be approximated as a constrained maximization problem such that PGD-based iterative methods can be directly applied. In the model-based setting, we show that the PGD updates can be efficiently performed using back-propagation. In the model-free setting, we propose a finite-difference PGD estimate which is general and only requires a black-box simulator for generating the trajectories of the closed-loop system given any initial state. Finally, we demonstrate the scalability and generality of our analysis tool on several numerical examples with large state dimensions or complex image observations.

show abstract

Do Differentiable Simulators Give Better Policy Gradients?

Cited by 3 publications

References 0 publications

New subspace minimization conjugate gradient methods based on regularization model for unconstrained optimization

New subspace minimization conjugate gradient methods based on regularization model for unconstrained optimization

Rethinking Optimization with Differentiable Simulation from a Global Perspective

Revisiting PGD Attacks for Stability Analysis of High-Dimensional Nonlinear Systems and Perception-Based Control

Contact Info

Product

Resources

About