Distributional and hierarchical reinforcement learning for physical systems with noisy state observations and exogenous perturbations

Park, Jehyun; Choi, Jongeun; Nah, Sungjae; Kim, Do-Hee

doi:10.1016/j.engappai.2023.106465

Cited by 3 publications

(3 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hierarchical reinforcement learning is a popular research area in the field of reinforcement learning that addresses the challenge of sparse rewards [74]. The main focus of this area is to design hierarchical structures that can effectively capture complex and abstracted decision processes in an agent.…”

Section: Hierarchical Reinforcement Learningmentioning

confidence: 99%

How to Design Reinforcement Learning Methods for the Edge: An Integrated Approach toward Intelligent Decision Making

Wu,

Zhang,

Miao

et al. 2024

Electronics

View full text Add to dashboard Cite

Extensive research has been carried out on reinforcement learning methods. The core idea of reinforcement learning is to learn methods by means of trial and error, and it has been successfully applied to robotics, autonomous driving, gaming, healthcare, resource management, and other fields. However, when building reinforcement learning solutions at the edge, not only are there the challenges of data-hungry and insufficient computational resources but also there is the difficulty of a single reinforcement learning method to meet the requirements of the model in terms of efficiency, generalization, robustness, and so on. These solutions rely on expert knowledge for the design of edge-side integrated reinforcement learning methods, and they lack high-level system architecture design to support their wider generalization and application. Therefore, in this paper, instead of surveying reinforcement learning systems, we survey the most commonly used options for each part of the architecture from the point of view of integrated application. We present the characteristics of traditional reinforcement learning in several aspects and design a corresponding integration framework based on them. In this process, we show a complete primer on the design of reinforcement learning architectures while also demonstrating the flexibility of the various parts of the architecture to be adapted to the characteristics of different edge tasks. Overall, reinforcement learning has become an important tool in intelligent decision making, but it still faces many challenges in the practical application in edge computing. The aim of this paper is to provide researchers and practitioners with a new, integrated perspective to better understand and apply reinforcement learning in edge decision-making tasks.

show abstract

Section: Hierarchical Reinforcement Learningmentioning

confidence: 99%

How to Design Reinforcement Learning Methods for the Edge: An Integrated Approach toward Intelligent Decision Making

Wu,

Zhang,

Miao

et al. 2024

Electronics

View full text Add to dashboard Cite

show abstract

“…Moreover, the comprehensive evaluation of RL algorithms, particularly those harnessing state-of-the-art deep learning innovations, is essential for assessing their efficiency and robustness in noisy conditions (Sun et al, 2023;Park et al, 2023). This would allow for a more thorough evaluation of algorithmic performance and resilience in the face of noise.…”

Section: Neurobiological Foundations Of Rl Algorithms In Noisy Enviro...mentioning

confidence: 99%

“…The inherent structure of RL methods is challenged by issues such as sample efficiency, scalability, generalization, and robustness. One of the most significant challenges for RL in real-world applications is its sensitivity to noisy observations, as real-world sensors invariably contain some level of noise (Park et al, 2023;Sun et al, 2023;Dulac-Arnold et al, 2021). Value-based approaches like Q-learning (Watkins and Dayan, 1992) are vulnerable to noisy observations (Fox et al, 2015;Moreno et al, 2006).…”

Section: Introductionmentioning

confidence: 99%

Exploring the Noise Resilience of Successor Features and Predecessor Features Algorithms in One and Two-Dimensional Environments

Lee

2024

Preprint

View full text Add to dashboard Cite

Based on the predictive map theory of spatial learning in animals, this study delves into the dynamics of Successor Feature (SF) and Predecessor Feature (PF) algorithms within noisy environments. Utilizing Q-learning and Q($\lambda$) learning as benchmarks for comparative analysis, our investigation yielded unexpected outcomes. Contrary to prevailing expectations and previous literature where PF demonstrated superior performance, our findings reveal that in noisy environments, PF did not surpass SF. In a one-dimensional grid world, SF exhibited superior adaptability, maintaining robust performance across varying noise levels. This trend of diminishing performance with increasing noise was consistent across all examined algorithms, indicating a linear degradation pattern.The scenario shifted in a two-dimensional grid world, where the impact of noise on algorithm performance demonstrated a non-linear relationship, influenced by the $\lambda$ parameter of the eligibility trace. This complexity suggests that the interaction between noise and algorithm efficacy is tied to the environmental dimensionality and specific algorithmic parameters.Furthermore, this research contributes to the bridging discourse between computational neuroscience and reinforcement learning (RL), exploring the neurobiological parallels of SF and PF learning in spatial navigation. Despite the unforeseen performance trends, the findings enrich our comprehension of the strengths and weaknesses inherent in RL algorithms. This knowledge is pivotal for advancing applications in robotics, gaming AI, and autonomous vehicle navigation, underscoring the imperative for continued exploration into how RL algorithms process and learn from noisy inputs.

show abstract

Investigating Transfer Learning in Noisy Environments: A Study of Predecessor and Successor Features in Spatial Learning Using a T-Maze

Seo,

Lee

2024

Sensors

View full text Add to dashboard Cite

In this study, we investigate the adaptability of artificial agents within a noisy T-maze that use Markov decision processes (MDPs) and successor feature (SF) and predecessor feature (PF) learning algorithms. Our focus is on quantifying how varying the hyperparameters, specifically the reward learning rate (αr) and the eligibility trace decay rate (λ), can enhance their adaptability. Adaptation is evaluated by analyzing the hyperparameters of cumulative reward, step length, adaptation rate, and adaptation step length and the relationships between them using Spearman’s correlation tests and linear regression. Our findings reveal that an αr of 0.9 consistently yields superior adaptation across all metrics at a noise level of 0.05. However, the optimal setting for λ varies by metric and context. In discussing these results, we emphasize the critical role of hyperparameter optimization in refining the performance and transfer learning efficacy of learning algorithms. This research advances our understanding of the functionality of PF and SF algorithms, particularly in navigating the inherent uncertainty of transfer learning tasks. By offering insights into the optimal hyperparameter configurations, this study contributes to the development of more adaptive and robust learning algorithms, paving the way for future explorations in artificial intelligence and neuroscience.

show abstract

Distributional and hierarchical reinforcement learning for physical systems with noisy state observations and exogenous perturbations

Cited by 3 publications

References 17 publications

How to Design Reinforcement Learning Methods for the Edge: An Integrated Approach toward Intelligent Decision Making

How to Design Reinforcement Learning Methods for the Edge: An Integrated Approach toward Intelligent Decision Making

Exploring the Noise Resilience of Successor Features and Predecessor Features Algorithms in One and Two-Dimensional Environments

Investigating Transfer Learning in Noisy Environments: A Study of Predecessor and Successor Features in Spatial Learning Using a T-Maze

Contact Info

Product

Resources

About