Integrated guidance for Mars entry and powered descent using reinforcement learning and pseudospectral method

Jiang, Xiuqiang; Li, Shuang; Furfaro, Roberto

doi:10.1016/j.actaastro.2018.12.033

Cited by 35 publications

(12 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Then, combined with the methods of fuzzy reinforcement learning (FRL) (Jouffe, 1998) and proximal policy optimization (PPO) (Schulman et al, 2017), a repeated learning process can be implemented which will converge to the output network under the consideration of possible interferences and uncertainties in different scenarios. During a landing phase, the algorithm outputs control commands in real time according to the current state (Furfaro and Linares, 2017;Gaudet et al, 2018;Jiang et al, 2018).…”

Section: Learning-based Methodsmentioning

confidence: 99%

“…However, the ZEM/ZEV method can derive only the guidance command when omitting the aerodynamic forces. Recently, the reinforcement learning based guidance algorithm has also been adopted to enhance the robustness of the ZEM/ZEV algorithms (Furfaro and Linares, 2017;Gaudet et al, 2018;Jiang et al, 2018), but its effectiveness is affected by the training samples, and it is not adaptable to model uncertainties.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Survey of autonomous guidance methods for powered planetary landing

Song

Theil

Seelbinder

et al. 2020

Front Inform Technol Electron Eng

View full text Add to dashboard Cite

This paper summarizes the autonomous guidance methods (AGMs) for pinpoint soft landing on celestial surfaces. We first review the development of powered descent guidance methods, focusing on their contributions for dealing with constraints and enhancing computational efficiency. With the increasing demand for reusable launchers and more scientific returns from space exploration, pinpoint soft landing has become a basic requirement. Unlike the kilometer-level precision for previous activities, the position accuracy of future planetary landers is within tens of meters of a target respecting all constraints of velocity and attitude, which is a very difficult task and arouses renewed interest in AGMs. This paper states the generalized three-and six-degree-of-freedom optimization problems in the powered descent phase and compares the features of three typical scenarios, i.e., the lunar, Mars, and Earth landing. On this basis, the paper details the characteristics and adaptability of AGMs by comparing aspects of analytical guidance methods, numerical optimization algorithms, and learning-based methods, and discusses the convexification treatment and solution strategies for non-convex problems. Three key issues related to AGM application, including physical feasibility, model accuracy, and real-time performance, are presented afterward for discussion. Many space organizations, such as those in the United States, China, France, Germany, and Japan, have also developed free-flying demonstrators to carry out related research. The guidance methods which have been tested on these demonstrators are briefly introduced at the end of the paper.

show abstract

Section: Learning-based Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Survey of autonomous guidance methods for powered planetary landing

Song

Theil

Seelbinder

et al. 2020

Front Inform Technol Electron Eng

View full text Add to dashboard Cite

show abstract

“…Instead, control methods using AI will most often give statistical guarantees on performance when subject to specified uncertainties. One example of RL applied to powered descent is shown in Jiang et al (2019), which also incorporates pseudospectral optimal control methods. In this case, the RL controller is used to determine the 'handover' point at which the controller switches from entry-phase to powered descent-phase.…”

Section: Related Workmentioning

confidence: 99%

Improving the efficiency of reinforcement learning for a spacecraft powered descent with Q-learning

Wilson

Riccardi

2021

Optim Eng

View full text Add to dashboard Cite

Reinforcement learning entails many intuitive and useful approaches to solving various problems. Its main premise is to learn how to complete tasks by interacting with the environment and observing which actions are more optimal with respect to a reward signal. Methods from reinforcement learning have long been applied in aerospace and have more recently seen renewed interest in space applications. Problems in spacecraft control can benefit from the use of intelligent techniques when faced with significant uncertainties—as is common for space environments. Solving these control problems using reinforcement learning remains a challenge partly due to long training times and sensitivity in performance to hyperparameters which require careful tuning. In this work we seek to address both issues for a sample spacecraft control problem. To reduce training times compared to other approaches, we simplify the problem by discretising the action space and use a data-efficient algorithm to train the agent. Furthermore, we employ an automated approach to hyperparameter selection which optimises for a specified performance metric. Our approach is tested on a 3-DOF powered descent problem with uncertainties in the initial conditions. We run experiments with two different problem formulations—using a ‘shaped’ state representation to guide the agent and also a ‘raw’ state representation with unprocessed values of position, velocity and mass. The results show that an agent can learn a near-optimal policy efficiently by appropriately defining the action-space and state-space. Using the raw state representation led to ‘reward-hacking’ and poor performance, which highlights the importance of the problem and state-space formulation in successfully training reinforcement learning agents. In addition, we show that the optimal hyperparameters can vary significantly based on the choice of loss function. Using two sets of hyperparameters optimised for different loss functions, we demonstrate that in both cases the agent can find near-optimal policies with comparable performance to previously applied methods.

show abstract

“…Наконец, отметим еще один крупный класс работ, использующих методы обучения с подкреплением, это работы американских специалистов Р. Фурфаро (Roberto Furfaro) и Б. Гаудета (Brian Gaudet) [20,21,22,23,24]. Все эти работы посвящены задаче адаптивного управления аппаратом во время спуска на поверхность Марса или другого небесного тела.…”

Section: Introductionunclassified