An On-Line Planner for POMDPs with Large Discrete Action Space: A Quantile-Based Approach

Wang, Erli; Kurniawati, Hanna; Kroese, Dirk P.

doi:10.1609/icaps.v28i1.13906

Cited by 7 publications

(3 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Cross-Entropy method has been used in several algorithms for solving POMDPs and MDPs (the fully observable variant of POMDPs). Several of them consider discrete action spaces (Mannor, Rubinstein, and Gat 2003;Oliehoek, Kooij, and Vlassis 2008;Wang, Kurniawati, and Kroese 2018), while we consider POMDPs with continuous action spaces. Omidshafiei et al (2016) consider continuous actions spaces, but the optimization is carried out over a finite policy space.…”

Section: Related Workmentioning

confidence: 99%

A Surprisingly Simple Continuous-Action POMDP Solver: Lazy Cross-Entropy Search Over Policy Trees

Hoerger,

Kurniawati,

Kroese

et al. 2024

AAAI

View full text Add to dashboard Cite

The Partially Observable Markov Decision Process (POMDP) provides a principled framework for decision making in stochastic partially observable environments. However, computing good solutions for problems with continuous action spaces remains challenging. To ease this challenge, we propose a simple online POMDP solver, called Lazy Cross-Entropy Search Over Policy Trees (LCEOPT). At each planning step, our method uses a novel lazy Cross-Entropy method to search the space of policy trees, which provide a simple policy representation. Specifically, we maintain a distribution on promising finite-horizon policy trees. The distribution is iteratively updated by sampling policies, evaluating them via Monte Carlo simulation, and refitting them to the top-performing ones. Our method is lazy in the sense that it exploits the policy tree representation to avoid redundant computations in policy sampling, evaluation, and distribution update. This leads to computational savings of up to two orders of magnitude. Our LCEOPT is surprisingly simple as compared to existing state-of-the-art methods, yet empirically outperforms them on several continuous-action POMDP problems, particularly for problems with higher-dimensional action spaces.

show abstract

Section: Related Workmentioning

confidence: 99%

A Surprisingly Simple Continuous-Action POMDP Solver: Lazy Cross-Entropy Search Over Policy Trees

Hoerger,

Kurniawati,

Kroese

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…More recently, MCTS-based methods have been extended to handle problems with large or continuous action and observation spaces. QBASE (Wang et al, 2018) can handle problems with up to 10,000 discrete actions by extending the cross-entropy method (Rubinstein and Kroese, 2013) to MCTS. GPS-ABT (Seiler et al, 2015) uses General Pattern Search, a derivative-free optimization method to search for local optima in continuous action spaces.…”

Section: Related Pomdp Solversmentioning

confidence: 99%

Multilevel Monte Carlo for solving POMDPs on-line

Hoerger

Kurniawati

Elfes

2022

The International Journal of Robotics Research

Self Cite

View full text Add to dashboard Cite

Planning under partial observability is essential for autonomous robots. A principled way to address such planning problems is the Partially Observable Markov Decision Process (POMDP). Although solving POMDPs is computationally intractable, substantial advancements have been achieved in developing approximate POMDP solvers in the past two decades. However, computing robust solutions for systems with complex dynamics remains challenging. Most on-line solvers rely on a large number of forward simulations and standard Monte Carlo methods to compute the expected outcomes of actions the robot can perform. For systems with complex dynamics, for example, those with non-linear dynamics that admit no closed-form solution, even a single forward simulation can be prohibitively expensive. Of course, this issue exacerbates for problems with long planning horizons. This paper aims to alleviate the above difficulty. To this end, we propose a new on-line POMDP solver, called Multilevel POMDP Planner (MLPP), that combines the commonly known Monte-Carlo-Tree-Search with the concept of Multilevel Monte Carlo to speed up our capability in generating approximately optimal solutions for POMDPs with complex dynamics. Experiments on four different problems involving torque control, navigation and grasping indicate that MLPP substantially outperforms state-of-the-art POMDP solvers.

show abstract

“…Methods have been proposed to alleviate this issue (Seiler, Kurniawati, and Singh 2015), (Sunberg and Kochenderfer 2018). However, existing solvers can only perform well for problems with 3-4 continuous action spaces (Seiler, Kurniawati, and Singh 2015), while a method that can perform well for problems with 100,000 discrete actions was only recently proposed (Wang, Kurniawati, and Kroese 2018). To alleviate this issue, we resort to use a small discrete number of fixed increments and decrements of the joint angles as the action space.…”

Section: Introductionmentioning

confidence: 99%

POMDP-Based Candy Server:Lessons Learned from a Seven Day Demo

Hoerger

Song

Kurniawati

et al. 2019

ICAPS

Self Cite

View full text Add to dashboard Cite

An autonomous robot must decide a good strategy to achieve its long term goal, despite various types of uncertainty. The Partially Observable Markov Decision Processes (POMDPs) is a principled framework to address such a decision making problem. Despite the computational intractability of solving POMDPs, the past decade has seen substantial advancement in POMDP solvers. This paper presents our experience in enabling on-line POMDP solving to become the sole motion planner for a robot manipulation demo at IEEE SIMPAR and ICRA 2018. The demo scenario is a candy-serving robot: A 6-DOFs robot arm must pick-up a cup placed on a table by a user, use the cup to scoop candies from a box, and put the cup of candies back on the table. The average perception error is ∼3cm (≈ the radius of the cup), affecting the position of the cup and the surface level of the candies. This paper presents a strategy to alleviate the curse of history issue plaguing this scenario, the perception system and its integration with the planner, and lessons learned in enabling an online POMDP solver to become the sole motion planner of this entire task. The POMDP-based system were tested through a 7 days live demo at the two conferences. In this demo, 150 runs were attempted and 98% of them were successful. We also conducted further experiments to test the capability of our POMDP-based system when the environment is relatively cluttered by obstacles and when the user moves the cup while the robot tries to pick it up. In both cases, our POMDP-based system reaches a success rate of 90% and above.

show abstract

An On-Line Planner for POMDPs with Large Discrete Action Space: A Quantile-Based Approach

Cited by 7 publications

References 13 publications

A Surprisingly Simple Continuous-Action POMDP Solver: Lazy Cross-Entropy Search Over Policy Trees

A Surprisingly Simple Continuous-Action POMDP Solver: Lazy Cross-Entropy Search Over Policy Trees

Multilevel Monte Carlo for solving POMDPs on-line

POMDP-Based Candy Server:Lessons Learned from a Seven Day Demo

Contact Info

Product

Resources

About