Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework

Jin, Wanxin; Wang, Zhaoran; Yang, Zhuoran; Mou, Shaoshuai

doi:10.48550/arxiv.1912.12970

Cited by 4 publications

(3 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Analogously to the parameter fitting process for the 2OL-Eq and MinJerk models, we identify optimal values for the cost weights 𝜔 𝑣 , 𝜔 𝑓 , and 𝜔 𝑟 collected in the paramter vector Λ for each mean trajectory. Since these parameters only define the objective function 𝐽 (LQR) 𝑁 of the OCP, and the system matrices 𝐴 and 𝐵 are uniquely determined given the above fixed values of 𝑚, 𝜏 1 , and 𝜏 2 , the parameter fitting for the LQR can be regarded as an inverse optimal control problem [62]. 8 As usual, we use the bi-level approach described in Section 4.2, i.e., at each iteration of the parameter fitting method, the OCP subject to the parameter vector Λ is solved as described above.…”

Section: Results Of Parameter Fittingmentioning

confidence: 99%

Optimal Feedback Control for Modeling Human-Computer Interaction

Fischer,

Fleig,

Klar

et al. 2021

Preprint

View full text Add to dashboard Cite

Optimal feedback control (OFC) is a theory from the motor control literature that explains how humans move their body to achieve a certain goal, e.g., pointing with the finger. OFC is based on the assumption that humans aim to control their body optimally, within the constraints imposed by body, environment, and task. In this paper, we explain how this theory can be applied to understanding Human-Computer Interaction. We propose that in this case, the dynamics of the human body and computer can be interpreted as a single dynamical system. The state of this system is controlled by the user via muscle control signals, and estimated from observations. Between-trial variability arises from signal-dependent control noise and observation noise. We compare four different models from optimal control theory and evaluate to what degree these models can replicate movements in the case of mouse pointing. We introduce a procedure to identify parameters that best explain observed user behavior, and show how these parameters can be used to gain insights regarding user characteristics and preferences. We conclude that OFC presents a powerful framework for HCI to understand and simulate motion of the human body and of the interface on a moment by moment basis.CCS Concepts: • Human-centered computing → HCI theory, concepts and models.

show abstract

Section: Results Of Parameter Fittingmentioning

confidence: 99%

Optimal Feedback Control for Modeling Human-Computer Interaction

Fischer,

Fleig,

Klar

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…To inject an appropriate inductive bias into the modeling procedure, recent work shows the possibility of embedding differentiable optimization problems as layers in an end-to-end framework [1]. To differentiate through an optimization problem, one could either unroll the numerical computation [17,4] or implicitly differentiate the optimality conditions, such as the KKT conditions in quadratic programs (QP) [3], the Pontryagin minimum principle in optimal control problems [25], and the Euler-Lagrange equations in least-action problems [36]. As the solution to a VI problem can usually be characterized as a fixed-point equation via a projection operator, which is equivalent to a QP problem, our work is built on some results in [3,1].…”

Section: Contribution This Paper Provides a Unified Framework For Lea...mentioning

confidence: 99%

End-to-End Learning and Intervention in Games

Li,

Yu,

Nie

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

In a social system, the self-interest of agents can be detrimental to the collective good, sometimes leading to social dilemmas. To resolve such a conflict, a central designer may intervene by either redesigning the system or incentivizing the agents to change their behaviors. To be effective, the designer must anticipate how the agents react to the intervention, which is dictated by their often unknown payoff functions. Therefore, learning about the agents is a prerequisite for intervention. In this paper, we provide a unified framework for learning and intervention in games. We cast the equilibria of games as individual layers and integrate them into an endto-end optimization framework. To enable the backward propagation through the equilibria of games, we propose two approaches, respectively based on explicit and implicit differentiation. Specifically, we cast the equilibria as the solutions to variational inequalities (VIs). The explicit approach unrolls the projection method for solving VIs, while the implicit approach exploits the sensitivity of the solutions to VIs. At the core of both approaches is the differentiation through a projection operator. Moreover, we establish the correctness of both approaches and identify the conditions under which one approach is more desirable than the other. The analytical results are validated using several real-world problems.

show abstract

“…Unlike the majority of work in trajectory optimization and control, we use Pontryagin's maximum principle to learn global feedback policies instead of fitting single trajectories. Concurrently Jin et al (2019) explored differentiating through Pontryagin's maximum principle. However, they still considered the standard time discretizations, eg.…”

Section: Related Workmentioning

confidence: 99%

Faster Policy Learning with Continuous-Time Gradients

Ainsworth¹,

Lowrey²,

Thickstun³

et al. 2020

Preprint

View full text Add to dashboard Cite

We study the estimation of policy gradients for continuous-time systems with known dynamics. By reframing policy learning in continuous-time, we show that it is possible construct a more efficient and accurate gradient estimator. The standard back-propagation through time estimator (BPTT) computes exact gradients for a crude discretization of the continuous-time system. In contrast, we approximate continuous-time gradients in the original system. With the explicit goal of estimating continuous-time gradients, we are able to discretize adaptively and construct a more efficient policy gradient estimator which we call the Continuous-Time Policy Gradient (CTPG). We show that replacing BPTT policy gradients with more efficient CTPG estimates results in faster and more robust learning in a variety of control tasks and simulators.

show abstract

Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework

Cited by 4 publications

References 25 publications

Optimal Feedback Control for Modeling Human-Computer Interaction

Optimal Feedback Control for Modeling Human-Computer Interaction

End-to-End Learning and Intervention in Games

Faster Policy Learning with Continuous-Time Gradients

Contact Info

Product

Resources

About