Self-Supervised Online Reward Shaping in Sparse-Reward Environments

Memarian, Farzan; Goo, Wonjoon; Lioutikov, Rudolf; Topcu, Ufuk; Niekum, Scott

doi:10.1109/iros51168.2021.9636020

Cited by 23 publications

(6 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nevertheless, the reduced-order models and learning algorithms used in this study have potential improvements that can be made in future research, including (1) collecting more quantitative data on storm petrel pattering/seaanchoring to validate modeling approaches, (2) exploring new learning techniques, such as reward shaping [37,38], to enhance the interoperability of the AI 'black box' , and (3) using empirical data to measure the drag coefficient of objects that replicate a storm petrel's anatomy by fluid dynamic experiments to build a higher fidelity force model [39][40][41].…”

Section: Discussionmentioning

confidence: 99%

Exploring storm petrel pattering and sea-anchoring using deep reinforcement learning

Xue,

Han,

Klaassen van Oorschot

et al. 2023

Bioinspir. Biomim.

View full text Add to dashboard Cite

Developing hybrid aerial-aquatic vehicles that can interact with water surfaces while remaining aloft is valuable for various tasks, including ecological monitoring, water quality sampling, and search and rescue operations. Storm petrels are a group of pelagic seabirds that exhibit a unique locomotion pattern known as "pattering" or "sea-anchoring," which is hypothesized to support forward locomotion and/or stationary posture at the water surface. In this study, we use morphological measurements of three storm petrel species and aero/hydrodynamic models to develop a computational storm petrel model and interact it with a hybrid fluid environment. Using Deep Reinforcement Learning (DRL) algorithms, we find that the storm petrel model exhibits high maneuverability and stability under a wide range of constant wind velocities after training. We also verify in the simulation that the storm petrel can use its “pattering” or “sea-anchoring” behavior to achieve different biomechanical sub-tasks (e.g., weight support, forward locomotion, stabilization) and adapt it under different wind speeds and optimization objectives. Specifically, we observe an adjustment in storm petrel’s movement patterns as wind velocity increases and quantitively analyze its biomechanics underneath. Our results provide new insights into how storm petrels achieve efficient locomotion and dynamic stability at the air-water interface and adapt their behaviors to different wind velocities and tasks in open environments. Ultimately, our study will guide the design of next-generation biomimetic petrel-inspired robots for tasks requiring proximity to the water interface and efficiency.

show abstract

Section: Discussionmentioning

confidence: 99%

Exploring storm petrel pattering and sea-anchoring using deep reinforcement learning

Xue,

Han,

Klaassen van Oorschot

et al. 2023

Bioinspir. Biomim.

View full text Add to dashboard Cite

show abstract

“…Likewise, each reward component should be weighted optimally in a multi-objective DRL agent to achieve the desired outcome and faster convergence. Few studies [27], [28] use supervised reward shaping techniques to assist the sparse rewards setup, which is a different approach from ours, as we focus on predicting optimal weights for each reward component according to the current contextual information.…”

Section: Related Workmentioning

confidence: 99%

An ML-Aided Reinforcement Learning Approach for Challenging Vehicle Maneuvers

Selvaraj

Hegde

Amati

et al. 2023

IEEE Trans. Intell. Veh.

View full text Add to dashboard Cite

The richness of information generated by today's vehicles fosters the development of data-driven decision-making models, with the additional capability to account for the context in which vehicles operate. In this work, we focus on Adaptive Cruise Control (ACC) in the case of such challenging vehicle maneuvers as cut-in and cut-out, and leverages Deep Reinforcement Learning (DRL) and vehicle connectivity to develop a data-driven cooperative ACC application. Our DRL framework accounts for all the relevant factors, namely, passengers' safety and comfort as well as efficient road capacity usage, and it properly weights them through a two-layer learning approach. We evaluate and compare the performance of the proposed scheme against existing alternatives through the CoMoVe framework, which realistically represents vehicle dynamics, communication and traffic. The results, obtained in different real-world scenarios, show that our solution provides excellent vehicle stability, passengers' comfort, and traffic efficiency, and highlight the crucial role that vehicle connectivity can play in ACC. Notably, our DRL scheme improves the road usage efficiency by being inside the desired range of headway in cut-out and cut-in scenarios for 69% and 78% (resp.) of the time, whereas alternatives respect the desired range only for 15% and 45% (resp.) of the time. We also validate the proposed solution through a hardware-in-the-loop implementation, and demonstrate that it achieves similar performance to that obtained through the CoMoVe framework.

show abstract

“…However, such an approach can easily exploit poorly designed rewards, get stuck in local optima, and induce behavior that the designer did not intend. In contrast, goal-based sparse rewards are appealing because they do not suffer from the reward exploration problem (24). In addition, this small, simple set of rules has similarities with biological behaviors and is therefore applicable to animals with a very limited level of information processing (51).…”

Section: Reward Functionmentioning

confidence: 99%

“…To enhance the target-tracking capabilities of marine robots and our understanding of the ecosystem, a guidance system based on soft actor-critic (SAC) (23) deep RL algorithms has been developed. Whereas most of the attention in deep RL has focused on game theory [for example, to solve Atari games (24) or to master the game of Go (25)], the same principles can be used to solve path planning and trajectory optimization problems (Table 1). Previous studies have shown that aerial gliders can navigate atmospheric thermals autonomously (26), stratospheric Loon superpressure balloons have learned optimal control to maintain their position at multiple locations (27), and an RL agent has been trained to efficiently navigate in simulated vortical flow fields (28).…”

Section: Introductionmentioning

confidence: 99%

Dynamic robotic tracking of underwater targets using reinforcement learning

et al. 2023

View full text Add to dashboard Cite

To realize the potential of autonomous underwater robots that scale up our observational capacity in the ocean, new techniques are needed. Fleets of autonomous robots could be used to study complex marine systems and animals with either new imaging configurations or by tracking tagged animals to study their behavior. These activities can then inform and create new policies for community conservation. The role of animal connectivity via active movement of animals represents a major knowledge gap related to the distribution of deep ocean populations. Tracking underwater targets represents a major challenge for observing biological processes in situ, and methods to robustly respond to a changing environment during monitoring missions are needed. Analytical techniques for optimal sensor placement and path planning to locate underwater targets are not straightforward in such cases. The aim of this study was to investigate the use of reinforcement learning as a tool for range-only underwater target-tracking optimization, whose promising capabilities have been demonstrated in terrestrial scenarios. To evaluate its usefulness, a reinforcement learning method was implemented as a path planning system for an autonomous surface vehicle while tracking an underwater mobile target. A complete description of an open-source model, performance metrics in simulated environments, and evaluated algorithms based on more than 15 hours of at-sea field experiments are presented. These efforts demonstrate that deep reinforcement learning is a powerful approach that enhances the abilities of autonomous robots in the ocean and encourages the deployment of algorithms like these for monitoring marine biological systems in the future.

show abstract

Self-Supervised Online Reward Shaping in Sparse-Reward Environments

Cited by 23 publications

References 9 publications

Exploring storm petrel pattering and sea-anchoring using deep reinforcement learning

Exploring storm petrel pattering and sea-anchoring using deep reinforcement learning

An ML-Aided Reinforcement Learning Approach for Challenging Vehicle Maneuvers

Dynamic robotic tracking of underwater targets using reinforcement learning

Contact Info

Product

Resources

About