Understanding the Behavior of Reinforcement Learning Agents

Stork, Jörg; Zaefferer, Martin; Bartz-Beielstein, Thomas; Eiben, A. E.

doi:10.1007/978-3-030-63710-1_12

Cited by 4 publications

(15 citation statements)

References 13 publications

(20 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The BNET prototype was not systematically tuned for optimal parameter settings due to the high computational effort. The used parameter setup is based on preliminary tests and CGP-ANN or SMB-NE related publications [28,29]. The baseline algorithms parameter were also improved (from the stable baseline default settings) based on preliminary results for each problem at hand, e.g., the reward discount parameter gamma and the learning rate.…”

Section: Methodsmentioning

confidence: 99%

“…Here, one challenge is the appropriate definition of the state set S, as it has a considerable influence on the distance. We rely on a selection approach presented in [29], where both policies' stored states are combined for each pairwise distance calculation. If a new target network's fitness without stored states needs to be predicted, the reference policies' stored states are applied as reference input.…”

Section: Optimization By Behavior Surrogatesmentioning

confidence: 99%

See 1 more Smart Citation

Behavior-based neuroevolutionary training in reinforcement learning

Stork

Zaefferer

Eisler

et al. 2021

Proceedings of the Genetic and Evolutionary Computation Conference Companion

Self Cite

View full text Add to dashboard Cite

In addition to their undisputed success in solving classical optimization problems, neuroevolutionary and population-based algorithms have become an alternative to standard reinforcement learning methods. However, evolutionary methods often lack the sample efficiency of standard value-based methods that leverage gathered state and value experience. If reinforcement learning for real-world problems with significant resource cost is considered, sample efficiency is essential. The enhancement of evolutionary algorithms with experience exploiting methods is thus desired and promises valuable insights. This work presents a hybrid algorithm that combines topology-changing neuroevolutionary optimization with value-based reinforcement learning. We illustrate how the behavior of policies can be used to create distance and loss functions, which benefit from stored experiences and calculated state values. They allow us to model behavior and perform a directed search in the behavior space by gradient-free evolutionary algorithms and surrogate-based optimization. For this purpose, we consolidate different methods to generate and optimize agent policies, creating a diverse population. We exemplify the performance of our algorithm on standard benchmarks and a purpose-built real-world problem. Our results indicate that combining methods can enhance the sample efficiency and learning speed for evolutionary approaches.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Optimization By Behavior Surrogatesmentioning

confidence: 99%

Behavior-based neuroevolutionary training in reinforcement learning

Stork

Zaefferer

Eisler

et al. 2021

Proceedings of the Genetic and Evolutionary Computation Conference Companion

Self Cite

View full text Add to dashboard Cite

show abstract

“…This notion of behavior, with slight modifications, has appeared in several papers in the Reinforcement Learning literature [23][24][25][26]. At least one existing work uses this notion of behavior in Novelty Search [23].…”

Section: Primitive Behaviormentioning

confidence: 99%

“…Another [24] uses it for optimization with an algorithm other than Novelty Search. [23,25,26] weight the constituent distances (i.e. w s is not constant), and [25] uses primitive behavior to study the relationship between behavior and reward.…”

Section: Primitive Behaviormentioning

confidence: 99%

See 1 more Smart Citation

Agent Spaces

Raisbeck¹,

Allen²,

Lee³

2021

Preprint

View full text Add to dashboard Cite

Exploration is one of the most important tasks in Reinforcement Learning, but it is not well-defined beyond finite problems in the Dynamic Programming paradigm (see Subsection 2.4). We provide a reinterpretation of exploration which can be applied to any online learning method.We come to this definition by approaching exploration from a new direction. After finding that concepts of exploration created to solve simple Markov decision processes with Dynamic Programming are no longer broadly applicable, we reexamine exploration. Instead of extending the ends of dynamic exploration procedures, we extend their means. That is, rather than repeatedly sampling every state-action pair possible in a process, we define the act of modifying an agent to itself be explorative. The resulting definition of exploration can be applied in infinite problems and non-dynamic learning methods, which the dynamic notion of exploration cannot tolerate.To understand the way that modifications of an agent affect learning, we describe a novel structure on the set of agents: a collection of distances (see footnote 7) da, a ∈ A, which represent the perspectives of each agent possible in the process. Using these distances, we define a topology and show that many important structures in Reinforcement Learning are well behaved under the topology induced by convergence in the agent space.

show abstract

Explainable reinforcement learning (XRL): a systematic literature review and taxonomy

Bekkemoen

2023

Mach Learn

View full text Add to dashboard Cite

In recent years, reinforcement learning (RL) systems have shown impressive performance and remarkable achievements. Many achievements can be attributed to combining RL with deep learning. However, those systems lack explainability, which refers to our understanding of the system’s decision-making process. In response to this challenge, the new explainable RL (XRL) field has emerged and grown rapidly to help us understand RL systems. This systematic literature review aims to give a unified view of the field by reviewing ten existing XRL literature reviews and 189 XRL studies from the past five years. Furthermore, we seek to organize these studies into a new taxonomy, discuss each area in detail, and draw connections between methods and stakeholder questions (e.g., “how can I get the agent to do _?”). Finally, we look at the research trends in XRL, recommend XRL methods, and present some exciting research directions for future research. We hope stakeholders, such as RL researchers and practitioners, will utilize this literature review as a comprehensive resource to overview existing state-of-the-art XRL methods. Additionally, we strive to help find research gaps and quickly identify methods that answer stakeholder questions.

show abstract

Understanding the Behavior of Reinforcement Learning Agents

Cited by 4 publications

References 13 publications

Behavior-based neuroevolutionary training in reinforcement learning

Behavior-based neuroevolutionary training in reinforcement learning

Agent Spaces

Explainable reinforcement learning (XRL): a systematic literature review and taxonomy

Contact Info

Product

Resources

About