Hierarchical Reinforcement-Learning for Real-Time Scheduling of Agile Satellites

Ren, Lili; Ning, Xin; Li, Jiayin

doi:10.1109/access.2020.3040748

Cited by 13 publications

(8 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hierarchical reinforcement learning uses the decomposition of objectives into sub-objectives, which can effectively reduce the solution space and is an effective method for solving large-scale constellation decision-making problems [ 30 ]. Ren et al proposed a hierarchical reinforcement-learning algorithm based on Q-learning for response speed and stability problems in randomly occurring urgent tasks [ 31 ]. Zhao et al proposed a two-stage neural network combinatorial optimization method based on DDPG to solve the problem of temporal task assignments in dynamic environments [ 32 ].…”

Section: Introductionmentioning

confidence: 99%

An LEO Constellation Early Warning System Decision-Making Method Based on Hierarchical Reinforcement Learning

Cheng

Wei

Sun

et al. 2023

Sensors

View full text Add to dashboard Cite

The cooperative positioning problem of hypersonic vehicles regarding LEO constellations is the focus of this research study on space-based early warning systems. A hypersonic vehicle is highly maneuverable, and its trajectory is uncertain. New challenges are posed for the cooperative positioning capability of the constellation. In recent years, breakthroughs in artificial intelligence technology have provided new avenues for collaborative multi-satellite intelligent autonomous decision-making technology. This paper addresses the problem of multi-satellite cooperative geometric positioning for hypersonic glide vehicles (HGVs) by the LEO-constellation-tracking system. To exploit the inherent advantages of hierarchical reinforcement learning in intelligent decision making while satisfying the constraints of cooperative observations, an autonomous intelligent decision-making algorithm for satellites that incorporates a hierarchical proximal policy optimization with random hill climbing (MAPPO-RHC) is designed. On the one hand, hierarchical decision making is used to reduce the solution space; on the other hand, it is used to maximize the global reward and to uniformly distribute satellite resources. The single-satellite local search method improves the capability of the decision-making algorithm to search the solution space based on the decision-making results of the hierarchical proximal policy-optimization algorithm, combining both random hill climbing and heuristic methods. Finally, the MAPPO-RHC algorithm’s coverage and positioning accuracy performance is simulated and analyzed in two different scenarios and compared with four intelligent satellite decision-making algorithms that have been studied in recent years. From the simulation results, the decision-making results of the MAPPO-RHC algorithm can obtain more balanced resource allocations and higher geometric positioning accuracy. Thus, it is concluded that the MAPPO-RHC algorithm provides a feasible solution for the real-time decision-making problem of the LEO constellation early warning system.

show abstract

Section: Introductionmentioning

confidence: 99%

An LEO Constellation Early Warning System Decision-Making Method Based on Hierarchical Reinforcement Learning

Cheng

Wei

Sun

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…However, this method is also based on supervised learning data and has limited applicability as well as problem applicability as it is also a kind of supervised learning. Ren [12] proposed a hierarchical reinforcement learning method to improve the response speed and stability of autonomous satellites to urgent tasks. The base layer is used to learn and train the network, while the top layer uses the network to assign tasks.…”

Section: Introductionmentioning

confidence: 99%

Modeling and Solving for Multi-Satellite Cooperative Task Allocation Problem Based on Genetic Programming Method

Yang

Xing

et al. 2022

Mathematics

View full text Add to dashboard Cite

The past decade has seen an increase in the number of satellites in orbit and in highly dynamic satellite requests, making the control by ground stations inefficient. The traditional management composed of ground planning with separate onboard execution is seriously lagging in response to dynamically incoming tasks. To meet the demand for the real-time response to emergent events, a multi-autonomous-satellite system with a central-distributed collaborative architecture was formulated by an integer programming model. Based on the structure, evolutionary rules were proposed to solve this problem by the use of sequence solution construction and a constructed heuristic method based on gene expression programming evolution. First, the features of the problem are extracted based on domain knowledge, then, the problem-solving rules are evolved by gene expression programming. The simulation results reflect that the evolutionary rule completely surpasses the three types of heuristic rules with adaptive mechanisms and achieves a solution effect close to meta-heuristic algorithms with a reasonably fast solving speed.

show abstract

“…Several hierarchical reinforcement approaches have been proposed to improve the performance of multi-UAV wireless networks [11]- [13]. In [11], to resolve the problem of limited data collection coverage of the backscatter sensor nodes, the hierarchical deep reinforcement learning (DRL) framework was proposed to extend the data collection coverage and minimize the total flight time of the rechargeable UAVs when performing data collecting missions.…”

Section: Introductionmentioning

confidence: 99%

“…The proposed h-DQN shows faster convergence, higher performance, and higher channel utilization than Q-learning for dynamic sensing (QADS) [14] or deep reinforcement learning for dynamic access (DRLDA) [15]. Additional, in [13], a hierarchical scheduling architecture with top-layer scheduling for satellite selection and foundation-layer precise scheduling for urgent tasks was introduced to solve the real-time earth observation satellite (EOS) scheduling problem. Here, Q-learning with an adaptive action selection strategy was proposed to solve the Markov decision process model more efficiently.…”

Section: Introductionmentioning

confidence: 99%

Optimal Frequency Reuse and Power Control in Multi-UAV Wireless Networks: Hierarchical Multi-Agent Reinforcement Learning Perspective

Lee

Lim

Chae³

et al. 2022

IEEE Access

View full text Add to dashboard Cite

To overcome the problems caused by the limited battery lifetime in multiple-unmanned aerial vehicle (UAV) wireless networks, we propose a hierarchical multi-agent reinforcement learning (RL) framework to maximize the energy efficiency (EE) of UAVs by finding the optimal frequency reuse factor and transmit power. The proposed algorithm consists of distributed inner-loop RL for transmit power control of the UAV terminal (UT) and centralized outer-loop RL for finding the optimal frequency reuse factor. Specifically, the proposed algorithm adjusts these two factors jointly to effectively mitigate intercell interference and reduce undesired transmit power consumption in multi-UAV wireless networks. We show that, for this reason, the proposed algorithm outperforms conventional algorithms, such as a random action algorithm with a fixed frequency reuse factor and a hierarchical multi-agent Q-learning algorithm with binary transmit power controls. Furthermore, even in the environment where UTs are continuously moving based on the mixed mobility model, we show that the proposed algorithm can find the best reward when compared to conventional algorithms.INDEX TERMS Unmanned aerial vehicle, optimal frequency reuse, transmit power control, energy efficiency, hierarchical multi-agent Q-learning, multi-UAV wireless network.

show abstract

Hierarchical Reinforcement-Learning for Real-Time Scheduling of Agile Satellites

Cited by 13 publications

References 28 publications

An LEO Constellation Early Warning System Decision-Making Method Based on Hierarchical Reinforcement Learning

An LEO Constellation Early Warning System Decision-Making Method Based on Hierarchical Reinforcement Learning

Modeling and Solving for Multi-Satellite Cooperative Task Allocation Problem Based on Genetic Programming Method

Optimal Frequency Reuse and Power Control in Multi-UAV Wireless Networks: Hierarchical Multi-Agent Reinforcement Learning Perspective

Contact Info

Product

Resources

About