Learning to Guide: Guidance Law Based on Deep Meta-Learning and Model Predictive Path Integral Control

Abstract

In this paper, we present a novel guidance scheme based on model-based deep reinforcement learning (RL) technique. With model-based deep RL method, a deep neural network is trained as a predictive model of guidance dynamics which is incorporated into a model predictive path integral (MPPI) control framework. However, the traditional MPPI framework assumes the actual environment similar to the training dataset for the deep neural network which is impractical in practice with different maneuvering of the target, other perturbations, and actuator failures. To address this problem, our method utilizes meta-learning technique to make the deep neural dynamics model adapt to such changes online. With this approach, we can alleviate the performance deterioration of standard MPPI control caused by the difference between the actual environment and training data. Then, a novel guidance law for a varying velocity interceptor intercepting maneuvering target with desired terminal impact angle under actuator failure is constructed based on the aforementioned techniques. The simulation and experiment results under different cases show the effectiveness and robustness of the proposed guidance law in achieving successful interceptions of maneuvering target. INDEX TERMS Missile guidance, model predictive control, meta-learning, deep reinforcement learning, impact angle constraint.

Range-Aware Impact Angle Guidance Law With Deep Reinforcement Meta-Learning

“…The neural network utilized is a multi-layer dense network with ReLU activations. This deep neural dynamic is verified in [7] to have a neglectable prediction error, which will make failure of proposed deep RL controller unlikely.…”

“…Model-based RL is also used in guidance law design. In [7], a novel adaptive intercept angle guidance law with deep meta-RL is proposed for missile with actuator failures. Our work differs with [7] in the tube-MPPI approach and range aware hyperbolic functions that enhance the guidance performance.…”

“…MPPI is firstly used on real hardware in aggressive driving of rally vehicles in Ref. [22], and is implemented in a wide range of real control tasks including complex robot manipulation [23], missile guidance [7] and so on. Many attempts have been made to robustifies MPPI method.…”

“…Then we can get the optimal control solution by minimizing the gap measured by Kullback-Leibler divergence: After expanding out the KL divergence, and analyzing the concave result, the optimal control sequence can be derived as follows. Since the optimal distribution ℚ * cannot be directly sampled, importance sampling technique is taken to get the sequence * = * ( ) , The temperature coefficient of this softmax distribution is designed to be as follows to normalize the cost function distribution as in [7] :…”

“…Ref. [7] utilizes MPPI method to solve the guidance problem under impact angle constraint. However, MPPI sometimes suffer from degradation of robustness when the deep neural dynamics differs with the real environment.…”