Autonomous navigation of stratospheric balloons using reinforcement learning

Bellemare, Marc G.; Candido, Salvatore; Castro, Pablo Samuel; Gong, Jun; Machado, Marlos C.; Moitra, Subhodeep; Ponda, Sameera S.; Wang, Ziyu

doi:10.1038/s41586-020-2939-8

Cited by 201 publications

(109 citation statements)

References 37 publications

Supporting

Mentioning

109

Contrasting

Order By: Relevance

“…controllers) in the physical world is more limited in comparison, due to difficulties in scaling parallel data collection, and higher variability in real-world data. Bellemare et al (2020) present a case study on autonomous balloon navigation, adopting a Q-learning approach, rather than model-based like ours. Akkaya et al (2019) use domain randomization in simulation to close the sim2real gap for a difficult dexterous manipulation task.…”

Section: Related Workmentioning

confidence: 99%

Machine Learning for Mechanical Ventilation Control

Suo

Zhang

Gradu

et al. 2021

Preprint

View full text Add to dashboard Cite

We consider the problem of controlling an invasive mechanical ventilator for pressure-controlled ventilation: a controller must let air in and out of a sedated patient’s lungs according to a trajectory of airway pressures specified by a clinician.Hand-tuned PID controllers and similar variants have comprised the industry standard for decades, yet can behave poorly by over- or under-shooting their target or oscillating rapidly.We consider a data-driven machine learning approach: First, we train a simulator based on data we collect from an artificial lung. Then, we train deep neural network controllers on these simulators. We show that our controllers are able to track target pressure waveforms significantly better than PID controllers.We further show that a learned controller generalizes across lungs with varying characteristics much more readily than PID controllers do.

show abstract

Section: Related Workmentioning

confidence: 99%

Machine Learning for Mechanical Ventilation Control

Suo

Zhang

Gradu

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…RL has been making large inroads in solving ever more complex problems. For instance, newer systems have managed to learn and optimize over systems for which they are not given explicit rules [4], [5], instrument controls [6], and building controls based on edge computing [7]. This would fundamentally change how we are able to design experimental field campaigns, speeding up scientific discovery by improving data quality and usefulness, and facilitating more effective usage of research funding.…”

Section: Narrativementioning

confidence: 99%

The Usage of Observing System Simulation Experiments and Reinforcement Learning to Optimize Experimental Design and Operation

Hardin

2021

View full text Add to dashboard Cite

The focal area is data acquisition and assimilation enabled by machine learning, in particular focusing on network design and optimization, as well as insights gained from complex data. Science ChallengeVarious organizations regularly conduct field campaigns across the globe designed to probe and improve atmospheric process understanding. However, these campaigns are mostly designed ad-hoc, and rely on anecdotes and knowledge of what was successful in previous campaigns. Such ad-hoc experiment design results in sub-optimal instrument siting and operation strategies. Inadequate targeted data collection of extreme weather is a major limitation to Earth and Environmental Systems Science Division (EESSD)'s goals of data model integration (4.5.3) [8], which limits the knowledge gained through the collected observations, holding back significant advances in predictability. While advances in instrument design are always chipping away at these limitations, we are not optimally utilizing the instrumentation we currently possess.

show abstract

“…Bellemare et al applied the distributional QR-DQN algorithm to realize stagnation point maintenance of high-altitude stratospheric balloons by determining the wind field use strategy [22]. The RL framework was applied to create a highperforming flight controller.…”

Section: Introductionmentioning

confidence: 99%

“…First, the basic Q-learning algorithm and the modified algorithms have very strong robustness and can be applied to a variety of tasks, such as video games [33] and quadrotor control [34]. Second, a deep Qnetwork algorithm has been used to guide the path of air vehicles and has achieved very good results [22] [35]. To our knowledge, this is the first time that the reinforcement learning method has been completely applied to HALE solarpowered aircraft flight trajectory planning.…”

Section: Introductionmentioning

confidence: 99%

Energy-Optimal Flight Strategy for Solar-Powered Aircraft Using Reinforcement Learning With Discrete Actions

2021

IEEE Access

View full text Add to dashboard Cite

The low efficiency of photovoltaic cells limits the energy absorption of high-altitude longendurance (HALE) solar-powered unmanned aircraft vehicles (UAVs), which dramatically weakens the capacity for long-endurance missions. Therefore, finding a method to extend the flight duration with finite solar energy drives extensive research. The present work introduces a method that applies a deep reinforcement learning (DRL) framework to generate an energy-optimized flight strategy for HALE solarpowered aircraft. The neural network controller is designed to realize autonomous flight navigation by giving commands of thrust, attack angle, and bank angle. A mission area with a radius of 5 km is assumed to test the RL controller performance. The simulation results show that the RL controller leads to a 28 % increase in the battery SoC after a 24-hour flight, which indicates that a controller based on the RL framework might be a potential method for solving the solar-powered UAV trajectory planning problem. Aiming to explore the applicability of the RL controller, a sustained flight test is implemented. The results show that a 39-day endurance flight is achieved by the RL controller, which is 50% higher than the base case with a steady flight trajectory.

show abstract

Autonomous navigation of stratospheric balloons using reinforcement learning

Cited by 201 publications

References 37 publications

Machine Learning for Mechanical Ventilation Control

Machine Learning for Mechanical Ventilation Control

The Usage of Observing System Simulation Experiments and Reinforcement Learning to Optimize Experimental Design and Operation

Energy-Optimal Flight Strategy for Solar-Powered Aircraft Using Reinforcement Learning With Discrete Actions

Contact Info

Product

Resources

About