Reinforcement Learning based Online Adaptive Flight Control for the Cessna Citation II(PH-LAB) Aircraft

Abstract

Online Adaptive Flight Control is interesting in the context of growing complexity of aircraft systems and their adaptability requirements to ensure safety. An Incremental Approximate Dynamic Programming (iADP) controller combines reinforcement learning methods, optimal control and online identified incremental model to achieve optimal adaptive control suitable for Nonlinear Time-Varying systems. The main contribution of this paper is twofold. Firstly, the iADP controller is designed to achieve automatic online rate control to track pilot commands via setpoints provided by the manual outer loop on Citation II Aircraft model. Secondly, to assess the controller performance in the presence of sensor dynamics and actuator dynamics, an analysis is carried out to identify causes of any performance degradation. The simulation results from iADP longitudinal control using full state feedback indicate that the discretization of sensor signals, sensor bias and transport delays did not have any significant effect on the controller performance or on the incremental model identification. However noisy signals and sensors delays are found to cause controller performance degradation. Appropriate filtering of signals resulted in better estimation of the incremental model subsequently improving the controller performance due to noisy signals. Control performance degradation due to sensor delays should be addressed in future before conducting flight tests on Citation II Aircraft.

Flight Testing Reinforcement Learning based Online Adaptive Flight Control Laws on CS-25 Class Aircraft

“…During [34]. This step-by-step clearance of different subsystems of iADP FCL ensured the analysis and rectification of any issues found with individual subsystems.…”

“…The following desirable features for the control method are defined as follows: a simpler control design strategy with less and interpretable learning parameters, ability to adapt in real-time to fast changing nonlinear dynamics of aircraft in case of a failure, an algorithm which is sample efficient [13] [14] and fast to converge on the conventional Flight Control Computer (FCC). In this regard, iADP based FCS [15] is a good choice amongst the RL based FCS with practical interests. This algorithm identifies a local linearized incremental model online, to estimate and minimize an infinite horizon quadratic cost-to-go, exclusively using the collected aircraft state data [16].…”

Soft Actor-Critic Deep Reinforcement Learning for Fault Tolerant Flight Control

“…A policy objective with an expectation over noise instead of an expectation over actions, making use of the Q-function as an approximation of expected future rewards and entropy, is introduced in Eq. (9).…”

“…Longitudinal control was proposed first by [7], and then the introduction of Incremental Dual Heuristic Programming (IDHP) demonstrated fault-tolerance on simple failure cases [8]. Using IDHP for coupled-dynamics body rate control, [9] showed that that a business jet aircraft could be controlled successfully. It was extended to altitude and attitude control with pre-tuned PID controllers in [10] and [11], yet as the longitudinal and lateral motions were decoupled severe failures cases where the coupling effects become too dominant were not tested.…”