Unobserved Is Not Equal to Non-existent: Using Gaussian Processes to Infer Immediate Rewards Across Contexts

Azizsoltani, Hamoon; Kim, Yeo Jin; Ausin, Markel Sanz; Barnes, Tiffany; Chi, Min

doi:10.24963/ijcai.2019/273

Cited by 12 publications

(6 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, our learned weightings over demonstrators could be seen as a form of attention [Zadeh et al, 2018]. The idea of learning shared features is inspired by both encoder sharing [Flet-Berliac and Preux, 2019] and uncertainty quantification [Azizsoltani et al, 2019;Brown et al, 2020]. Finally, our approach shares some of the similarities of Bayesian policy reuse [Rosman et al, 2016], by formulating the problem of policy selection as a Bayesian choice problem.…”

Section: Related Workmentioning

confidence: 99%

Bayesian Experience Reuse for Learning from Multiple Demonstrators

Gimelfarb

Sanner

Lee

2021

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

Learning from Demonstrations (LfD) is a powerful approach for incorporating advice from experts in the form of demonstrations. However, demonstrations often come from multiple sub-optimal experts with conflicting goals, rendering them difficult to incorporate effectively in online settings. To address this, we formulate a quadratic program whose solution yields an adaptive weighting over experts, that can be used to sample experts with relevant goals. In order to compare different source and target task goals safely, we model their uncertainty using normal-inverse-gamma priors, whose posteriors are learned from demonstrations using Bayesian neural networks with a shared encoder. Our resulting approach, which we call Bayesian Experience Reuse, can be applied for LfD in static and dynamic decision-making settings. We demonstrate its effectiveness for minimizing multi-modal functions, and optimizing a high-dimensional supply chain with cost uncertainty, where it is also shown to improve upon the performance of the demonstrators' policies.

show abstract

Section: Related Workmentioning

confidence: 99%

Bayesian Experience Reuse for Learning from Multiple Demonstrators

Gimelfarb

Sanner

Lee

2021

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

show abstract

“…This is an alternative prior method for inferring the immediate rewards from the delayed ones. Prior work has shown that InferGP works reasonably well in a wide range of offline RL tasks [8]. Our goal is to determine the efficacy of InferNet for offline RL tasks, when compared to immediate, delayed and InferGP rewards.…”

Section: Offline Rl Experimentsmentioning

confidence: 99%

“…Recently, several DRL approaches have been investigated for septic treatment, utilizing Electronic Health Records (EHRs). However, [17,18] only considered delayed rewards, while [8] leveraged the Gaussian process based immediate reward inference method, which is one of our baselines. Data: Our EHRs were collected from a large US healthcare system (July, 2013 to December, 2015).…”

Section: Rmsementioning

confidence: 99%

See 1 more Smart Citation

InferNet for Delayed Reinforcement Tasks: Addressing the Temporal Credit Assignment Problem

Ausin¹,

Azizsoltani²,

Ju³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The temporal Credit Assignment Problem (CAP) is a well-known and challenging task in AI. While Reinforcement Learning (RL), especially Deep RL, works well when immediate rewards are available, it can fail when only delayed rewards are available or when the reward function is noisy. In this work, we propose delegating the CAP to a Neural Network-based algorithm named InferNet that explicitly learns to infer the immediate rewards from the delayed rewards. The effectiveness of InferNet was evaluated on two online RL tasks: a simple GridWorld and 40 Atari games; and two offline RL tasks: GridWorld and a real-life Sepsis treatment task. For all tasks, the effectiveness of using the InferNet inferred rewards is compared against the immediate and the delayed rewards with two settings: with noisy rewards and without noise. Overall, our results show that the effectiveness of InferNet is robust against noisy reward functions and is an effective add-on mechanism for solving temporal CAP in a wide range of RL tasks, from classic RL simulation environments to a real-world RL problem and for both online and offline learning.

show abstract

“…In this paper, we show how sub-optimal demonstrators with conflicting goals can be ranked according to their alignment with the target task goal in a safe Bayesian manner, and reused directly by the target agent without learning auxiliary representations such as policies or value functions. Our approach is related to encoder sharing [18,34,48,49] and uncertainty quantification of rewards or policies [5,12,25,46], but our problem setting and methodology are different.…”

Section: Related Workmentioning

confidence: 99%

Bayesian Experience Reuse for Learning from Multiple Demonstrators

Gimelfarb¹,

Sanner²,

Lee³

2020

Preprint

View full text Add to dashboard Cite

Learning from demonstrations (LfD) improves the exploration efficiency of a learning agent by incorporating demonstrations from experts. However, demonstration data can often come from multiple experts with conflicting goals, making it difficult to incorporate safely and effectively in online settings. We address this problem in the static and dynamic optimization settings by modelling the uncertainty in source and target task functions using normal-inverse-gamma priors, whose corresponding posteriors are, respectively, learned from demonstrations and target data using Bayesian neural networks with shared features. We use this learned belief to derive a quadratic programming problem whose solution yields a probability distribution over the expert models. Finally, we propose Bayesian Experience Reuse (BERS) to sample demonstrations in accordance with this distribution and reuse them directly in new tasks. We demonstrate the effectiveness of this approach for static optimization of smooth functions, and transfer learning in a high-dimensional supply chain problem with cost uncertainty.

show abstract

Unobserved Is Not Equal to Non-existent: Using Gaussian Processes to Infer Immediate Rewards Across Contexts

Cited by 12 publications

References 11 publications

Bayesian Experience Reuse for Learning from Multiple Demonstrators

Bayesian Experience Reuse for Learning from Multiple Demonstrators

InferNet for Delayed Reinforcement Tasks: Addressing the Temporal Credit Assignment Problem

Bayesian Experience Reuse for Learning from Multiple Demonstrators

Contact Info

Product

Resources

About