Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning☆

Malekzadeh, Parvin; Hou, Ming; Plataniotis, Konstantinos N.

doi:10.1016/j.neucom.2023.01.076

Cited by 3 publications

(1 citation statement)

References 98 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The probabilistic neural network model is mainly used to estimate the stochasticity, and the ensemble model is used to estimate the cognitive uncertainty. In addition, some works (Malekzadeh et al, 2023;Wu et al, 2023b) reduced the impact of model prediction errors on model application through uncertainty perception.…”

Section: Literature Reviewmentioning

confidence: 99%

A Model-Based Reinforcement Learning Method with Conditional Variational Auto-Encoder

Zhu,

Ren,

et al. 2024

JDSIS

View full text Add to dashboard Cite

Model-based reinforcement learning can effectively improve the sample efficiency of reinforcement learning, but the environment model in this method has errors. The model errors can mislead the policy optimization, leading to suboptimal policy. To improve the generalization ability of the environment model, existing methods often use ensemble models or Bayesian models to build the environment model. However, these methods are computationally intensive and complex to update. Since the generated model can describe the stochastic nature of the environment, this paper proposes a model-based reinforcement learning method based on a conditional variational auto-encoder. In this paper, we use a conditional variational auto-encoder to learn task-related representations and apply the generative model to predict environmental changes. Considering the problem of multi-step error accumulation, model adaptation is utilized to minimize the difference between simulated and real data distributions. Furthermore, the experiments verified that the proposed method can learn task-relevant representations and accelerate policy learning.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

A Model-Based Reinforcement Learning Method with Conditional Variational Auto-Encoder

Zhu,

Ren,

et al. 2024

JDSIS

View full text Add to dashboard Cite

show abstract

A Robust Quantile Huber Loss with Interpretable Parameter Adjustment in Distributional Reinforcement Learning

Malekzadeh,

Plataniotis,

Poulos

et al. 2024

ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Active Inference and Reinforcement Learning: A Unified Inference on Continuous State and Action Spaces Under Partial Observability

Malekzadeh,

Plataniotis

2024

Neural Computation

View full text Add to dashboard Cite

Reinforcement learning (RL) has garnered significant attention for developing decision-making agents that aim to maximize rewards, specified by an external supervisor, within fully observable environments. However, many real-world problems involve partial or noisy observations, where agents cannot access complete and accurate information about the environment. These problems are commonly formulated as partially observable Markov decision processes (POMDPs). Previous studies have tackled RL in POMDPs by either incorporating the memory of past actions and observations or by inferring the true state of the environment from observed data. Nevertheless, aggregating observations and actions over time becomes impractical in problems with large decision-making time horizons and high-dimensional spaces. Furthermore, inference-based RL approaches often require many environmental samples to perform well, as they focus solely on reward maximization and neglect uncertainty in the inferred state. Active inference (AIF) is a framework naturally formulated in POMDPs and directs agents to select actions by minimizing a function called expected free energy (EFE). This supplies reward-maximizing (or exploitative) behavior, as in RL, with information-seeking (or exploratory) behavior. Despite this exploratory behavior of AIF, its use is limited to problems with small time horizons and discrete spaces due to the computational challenges associated with EFE. In this article, we propose a unified principle that establishes a theoretical connection between AIF and RL, enabling seamless integration of these two approaches and overcoming their limitations in continuous space POMDP settings. We substantiate our findings with rigorous theoretical analysis, providing novel perspectives for using AIF in designing and implementing artificial agents. Experimental results demonstrate the superior learning capabilities of our method compared to other alternative RL approaches in solving partially observable tasks with continuous spaces. Notably, our approach harnesses information-seeking exploration, enabling it to effectively solve reward-free problems and rendering explicit task reward design by an external supervisor optional.

show abstract

Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning☆

Cited by 3 publications

References 98 publications

A Model-Based Reinforcement Learning Method with Conditional Variational Auto-Encoder

A Model-Based Reinforcement Learning Method with Conditional Variational Auto-Encoder

A Robust Quantile Huber Loss with Interpretable Parameter Adjustment in Distributional Reinforcement Learning

Active Inference and Reinforcement Learning: A Unified Inference on Continuous State and Action Spaces Under Partial Observability

Contact Info

Product

Resources

About