Reinforcement learning in spacecraft control applications: Advances, prospects, and challenges

Tipaldi, Massimo; Iervolino, Raffaele; Massenio, Paolo Roberto

doi:10.1016/j.arcontrol.2022.07.004

Cited by 35 publications

(9 citation statements)

References 81 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The terminal velocity and altitude are set as V f = 3.2 Ma and h f À R 0 = 23 km, respectively. The initial heading angle ψ 0 is calculated as equation (15). The rest parameters intervals of initial states of the vehicle and no-fly zone position are summarized in Table 1.…”

Section: Reference Command Generation Using Supervised Learningmentioning

confidence: 99%

“…RL has been widely applied in various aerospace control fields, such as planetary landing, orbit transfer, attitude control, rendezvous and docking, and constellation orbital control, etc. 15 From existing research, deep reinforcement learning is utilized to solve powered descent and landing control of Mars or the Moon. [16][17][18] Stateof-the-art RL algorithms, Proximal Policy Optimization (PPO) and Twin Delayed Deep Deterministic Policy Gradient (TD3PG), are applied to obstacles avoidance of unmanned aerial vehicles (UAVs) 19 and missile, 20 respectively.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Integrated entry guidance with no-fly zone constraint using reinforcement learning and predictor-corrector technique

Gao,

Zhou,

Chen

2024

Proceedings of the Institution of Mechanical Engineers, Part G:

View full text Add to dashboard Cite

This paper presents an integrated entry guidance law for hypersonic glide vehicles with no-fly zone constraint. Existing methods that employ predictor-corrector technique and lateral guidance logic for both guidance and avoidance, may have limitations in response time and maneuverability when facing sudden threats, because the guidance cycle is limited by computational efficiency and the bank angle magnitude cannot be adjusted according to the urgency of the avoidance. To overcome these challenges, the proposed method divides the entry process into safe flight stages and no-fly zone avoidance stages, and introduces reinforcement learning to develop an intelligent avoidance strategy for the latter. This division reduces the complexity of the learning problem by restricting the state space and increases the applicability in the presence of multiple no-fly zones. The trained avoidance strategy can directly output continuous bank angle command through a single forward calculation, considering both guidance and avoidance requirements. This enables the full utilization of the vehicle’s maneuverability and supports a high command update frequency to effectively handle threats. Additionally, a network trained via supervised learning is employed to generate reference commands, accelerating the training convergence of reinforcement learning. Simulation results demonstrate the effectiveness of the proposed guidance law, highlighting its high computational efficiency, command stability, and robustness. Importantly, the approach offers convenience in extending to multiple no-fly zones and accommodating vast initial state spaces.

show abstract

Section: Reference Command Generation Using Supervised Learningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Integrated entry guidance with no-fly zone constraint using reinforcement learning and predictor-corrector technique

Gao,

Zhou,

Chen

2024

Proceedings of the Institution of Mechanical Engineers, Part G:

View full text Add to dashboard Cite

show abstract

“…Incorporating smart NCSs (SNCSs) into distributed satellite systems has been demonstrated to improve attitude control and reliability in communication. The smart approach to satellite attitude control is not new, [141][142][143] and comparatively more developed than its NCS counterparts. The dual design of control and network routing in these systems has increased cooperation capacity in simulations and robustness to network-induced errors.…”

Section: Advancements In Smart Ncs For Satellitesmentioning

confidence: 99%

Networked control systems and their applications to smart satellites: a survey

McCafferty-Leroux,

Wu,

Gadsden

2024

Sensors and Systems for Space Applications XVII

View full text Add to dashboard Cite

The advancement of Earth observation satellite research in past decades has demonstrated itself to be productive and increasingly important. Utilized for applications such as climate monitoring, communication, GPS, defense, and space research, our dependence on reliable satellite systems is ever-increasing. The success of satellites in these scenarios is fundamentally the result of its attitude determination system, consisting of control and estimation subsystems, which govern its sensors and actuators. For simple missions, attitude pose determination can be computed onboard the satellite. Typically, however, ground stations or other satellites (i.e. constellations) are involved in a satellite's operation, processing large amounts of data or complex control algorithms. This information and control cycle is enabled through the application of Networked Control Systems (NCS). The NCS uses a wireless network or communication system as the intermediate line of communication between plant, actuators, sensors, and other systems. This enables relatively fast communication and data transmittance over long distances, as well as the decentralization of navigation and control through system distribution. However, this method is vulnerable to various forms of time delay and packet loss, which ultimately affects the control performance of a satellite. It is demonstrated in literature that the effects of these NCS properties can be mitigated, increasing its viability, through various implementations of smart systems into the satellite framework. Using techniques such as neural networks and reinforcement learning, the satellite can perceive and act based on environmental information, while considering experiential memory and attention allocation. The following comprehensive survey discusses methods for improving the robustness of networked satellite systems from a smart systems perspective, providing an advanced foundation for these concepts.

show abstract

“…Therefore, machine learning-based EMSs have received extensive attention in recent years, with reinforcement learning (RL) and deep reinforcement learning (DRL) being the most widely studied. RL's main idea is to train a fully autonomous agent by interacting directly with its potential environment [31]. It is different from supervised and unsupervised machine learning which need static data during the training process.…”

Section: Literature Reviewmentioning

confidence: 99%

Soft Actor-Critic Algorithm-Based Energy Management Strategy for Plug-In Hybrid Electric Vehicle

Cui

2022

WEVJ

View full text Add to dashboard Cite

Plug-in hybrid electric vehicles (PHEVs) are equipped with more than one power source, providing additional degrees of freedom to meet the driver’s power demand. Therefore, the reasonable allocation of the power demand of each power source by the energy management strategy (EMS) to keep each power source operating in the efficiency zone is essential for improving fuel economy. This paper proposes a novel model-free EMS based on the soft actor-critic (SAC) algorithm with automatic entropy tuning to balance the optimization of energy efficiency with the adaptability of driving cycles. The maximum entropy framework is introduced into deep reinforcement learning-based energy management to improve the performance of exploring the internal combustion engine (ICE) as well as the electric motor (EM) efficiency interval. Specifically, the automatic entropy adjustment framework improves the adaptability to driving cycles. In addition, the simulation is verified by the data collected from the real vehicle. The results show that the introduction of automatic entropy adjustment can effectively improve vehicle equivalent fuel economy. Compared with traditional EMS, the proposed EMS can save energy by 4.37%. Moreover, it is able to adapt to different driving cycles and can keep the state of charge to the reference value.

show abstract

Reinforcement learning in spacecraft control applications: Advances, prospects, and challenges

Cited by 35 publications

References 81 publications

Integrated entry guidance with no-fly zone constraint using reinforcement learning and predictor-corrector technique

Integrated entry guidance with no-fly zone constraint using reinforcement learning and predictor-corrector technique

Networked control systems and their applications to smart satellites: a survey

Soft Actor-Critic Algorithm-Based Energy Management Strategy for Plug-In Hybrid Electric Vehicle

Contact Info

Product

Resources

About