Soft Actor-Critic Deep Reinforcement Learning for Fault Tolerant Flight Control

Dally, Killian; Kampen, E. van

doi:10.2514/6.2022-2078

Cited by 15 publications

(23 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These are DRL techniques that generally use a larger number of hidden layers in the ANNs of the RL architecture to be able to work in more complex environments and deal with large continuous state and action spaces. While the focus of this thesis is on the IDHP architecture applied to flight control, these models are also of interest as they are able to deal with more complex tasks like the direct control of the full 6 degrees of freedom of an aircraft [75]. However, due to the addition of hidden layers, these models require significantly longer training times and may have limited online learning capabilities.…”

Section: Other Model-free Reinforcement Learning Modelsmentioning

confidence: 99%

“…Thanks to the high sample efficiency of SAC, the controller development is much easier, and it has been able to outperform other state-of-the-art model-free algorithms like DDPG, PPO, and TD3. SAC has already been applied to the low-level control of a quadcopter [92], as well as for a large fixed-wing business jet [75]. The latter showed the promising capabilities of this method to develop robust controllers for CS-25 certified aircraft.…”

Section: Soft Actor-criticmentioning

confidence: 99%

“…To achieve this, it was decided to apply the same SHAP technique to a far more complex and complete RL controller like the SAC flight controller designed in [75]. This architecture is shown in Figure 5.…”

Section: Sac Controllermentioning

confidence: 99%

“…Therefore, it must be trained for very long times in an offline simulator to achieve a proper performance. This controller architecture was already pre-trained in [75] for a linear flight regime with a trim altitude of 2000 m and a trim velocity of 90 m/s. While this does not qualify as fulfilling the main objective of this thesis, which was to study the behaviour of RL flight controllers in non-linear flight regimes, it helps achieve the secondary objective of applying the SHAP technique to more complex RL architectures.…”

Section: Sac Controllermentioning

confidence: 99%

“…As observed, the performance of the controller is very good, as it is able to follow the altitude profile with very limited error, while maintaining a very small sideslip angle error and a decent performance following the roll angle reference. As shown in [75], this controller is also robust to many different kinds of failure, like loss of aileron effectiveness, the movement of the center of mass, or the loss of half of the horizontal tail. This simulation has a total of 120 seconds, and since the weights of this controller are constant, the whole time duration can be used to apply the SHAP analysis.…”

Section: Sac Controllermentioning

confidence: 99%

See 4 more Smart Citations

Explainable Artificial Intelligence Techniques for the Analysis of Reinforcement Learning in Non-Linear Flight Regimes

Pizarroso

Kampen

2023

AIAA SCITECH 2023 Forum

Self Cite

View full text Add to dashboard Cite

Reinforcement Learning is being increasingly applied to flight control tasks, with the objective of developing truly autonomous flying vehicles able to traverse highly variable environments and adapt to unknown situations or possible failures. However, the development of these increasingly complex models and algorithms further reduces our understanding of their inner workings. This can affect the safety and reliability of the algorithms, as it is difficult or even impossible to determine which are their failure characteristics and how they will react in situations never tested before. It is possible to remedy this lack of understading through the development of eXplainable Artifial Intelligence and eXplainable Reinforcement Learning methods like SHapley Additive Explanations. This tool is used to analyze the strategy learnt by an Actor-Critic Incremental Dual Heuristic Programming controller architecture when presented with a pitch rate or roll rate tracking task in non-linear flying conditions, such as at high angles of attack and large sideslip angles. This paper shows that, even in the non-linear flight regime, it is still more optimal for this controller architecture to learn quasi-linear control laws, although it seems to continuously modify the linear slope as if it was an extreme case of the gain scheduling technique.

show abstract

Section: Other Model-free Reinforcement Learning Modelsmentioning

confidence: 99%

Section: Soft Actor-criticmentioning

confidence: 99%

Section: Sac Controllermentioning

confidence: 99%

Section: Sac Controllermentioning

confidence: 99%

Section: Sac Controllermentioning

confidence: 99%

See 3 more Smart Citations

Explainable Artificial Intelligence Techniques for the Analysis of Reinforcement Learning in Non-Linear Flight Regimes

Pizarroso

Kampen

2023

AIAA SCITECH 2023 Forum

Self Cite

View full text Add to dashboard Cite

show abstract

Generalization and Computation for Policy Classes of Generative Adversarial Imitation Learning

Zhou

Zhang

Liu

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

A deep reinforcement learning control approach for high-performance aircraft

De Marco,

D’Onza,

Manfredi

2023

Nonlinear Dyn

View full text Add to dashboard Cite

This research introduces a flight controller for a high-performance aircraft, able to follow randomly generated sequences of waypoints, at varying altitudes, in various types of scenarios. The study assumes a publicly available six-degree-of-freedom (6-DoF) rigid aeroplane flight dynamics model of a military fighter jet. Consolidated results in artificial intelligence and deep reinforcement learning (DRL) research are used to demonstrate the capability to make certain manoeuvres AI-based fully automatic for a high-fidelity nonlinear model of a fixed-wing aircraft. This work investigates the use of a deep deterministic policy gradient (DDPG) controller agent, based on the successful applications of the same approach to other domains. In the particular application to flight control presented here, the effort has been focused on the design of a suitable reward function used to train the agent to achieve some given navigation tasks. The trained controller is successful on highly coupled manoeuvres, including rapid sequences of turns, at both low and high flight Mach numbers, in simulations reproducing a prey–chaser dogfight scenario. Robustness to sensor noise, atmospheric disturbances, different initial flight conditions and varying reference signal shapes is also demonstrated.

show abstract

Soft Actor-Critic Deep Reinforcement Learning for Fault Tolerant Flight Control

Cited by 15 publications

References 18 publications

Explainable Artificial Intelligence Techniques for the Analysis of Reinforcement Learning in Non-Linear Flight Regimes

Explainable Artificial Intelligence Techniques for the Analysis of Reinforcement Learning in Non-Linear Flight Regimes

Generalization and Computation for Policy Classes of Generative Adversarial Imitation Learning

A deep reinforcement learning control approach for high-performance aircraft

Contact Info

Product

Resources

About