Learning in POMDPs with Monte Carlo Tree Search

Abstract

The POMDP is a powerful framework for reasoning under outcome and information uncertainty, but constructing an accurate POMDP model is difficult. Bayes-Adaptive Partially Observable Markov Decision Processes (BA-POMDPs) extend POMDPs to allow the model to be learned during execution. BA-POMDPs are a Bayesian RL approach that, in principle, allows for an optimal trade-off between exploitation and exploration. Unfortunately, BA-POMDPs are currently impractical to solve for any non-trivial domain. In this paper, we extend the Monte-Carlo Tree Search method POMCP to BA-POMDPs and show that the resulting method, which we call BA-POMCP, is able to tackle problems that previous solution methods have been unable to solve. Additionally, we introduce several techniques that exploit the BA-POMDP structure to improve the efficiency of BA-POMCP along with proof of their convergence. * This is an extended version of a paper that was published at ICML'2017.

Active Perception in Adversarial Scenarios using Maximum Entropy Deep Reinforcement Learning

“…One restriction of this framework is that it assumes a fixed reactive probabilistic model of the opponent, implying stationary behavior without rationality. To mitigate performance degradation due to modeling uncertainty, existing approaches include Bayesian-Adaptive POMDP (BA-POMDP) [6], [7], robust POMDP [8], Chance-constrained POMDP (CC-POMDP) [9], and Interactive-POMDP (I-POMDP) [10].…”

“…BA-POMDP augments the state space with a state transition count and a state observation count variables as additional hidden states [6], [7]. It maintains a belief over the augmented state space, resulting in an optimal trade-off between model learning and reward collecting.…”

“…The soft-Q learning objective is to maximize the expected reward regularized by the entropy of the policy, t E bt,st,at∼ρπ γ t [r(b t , s t , a t ) + σH(π(•|b t , s t ))]. (7) The parameter σ controls the 'softness' of the policy. The nice interpretation of this objective function is maximizing accumulative reward while behaving as uncertain as possible, which is a desired property against an adversary.…”