A few lessons learned in reinforcement learning for quadcopter attitude control

Abstract

In the context of developing safe air transportation, our work is focused on understanding how Reinforcement Learning methods can improve the state of the art in traditional control, in nominal as well as non-nominal cases. The end goal is to train provably safe controllers, by improving both training and verification methods. In this paper, we explore this path for controlling the attitude of a quadcopter: we discuss theoretical as well as practical aspects of training neural nets for controlling a crazyflie 2.0 drone. In particular we describe thoroughly the choices in training algorithms, neural net architecture, hyperparameters, observation space etc. We also discuss the robustness of the obtained controllers, both to partial loss of power for one rotor and to wind gusts. Finally, we measure the performance of the approach by using a robust form of a signal temporal logic to quantitatively evaluate the vehicle's behavior. CCS CONCEPTS• Computer systems organization → Embedded and cyberphysical systems; Robotic control; • Software and its engineering → Formal methods; • Theory of computation → Modal and temporal logics; • Computing methodologies → Computational control theory; Reinforcement learning.

Successive Convexification for Optimal Control with Signal Temporal Logic Specifications

“…Such semantics have been defined to assess the robustness of systems to parameter or timing variations [6]. It has also been used recently [4] in the context of controller synthesis with reinforcement learning to provide a reward associated to the formalized STL properties. STL as part of the Model Predictive Control (MPC) design has also been studied in [18], in which the combinatorial nature of such specifications is preserved, and thus Mixed-Integer Linear Programming (MILP) has to be employed.…”

“…Note that in (5), (6), and (7), we still have non-differentiable min-or maxfunctions. One can certainly use the log-sum-exponential function in (4) to approximate these min-/max-functions, but this approach is usually not accurate around non-differentiable points, especially with only two variables, which is exactly the case we have on hand.…”