A POMDP Formulation of Proactive Learning

Wray, Kyle Hollins; Zilberstein, Shlomo

doi:10.1609/aaai.v30i1.10400

“…In addition, while we provide results with a practical problem (i.e., BCS), experimenting with other practical problems and datasets -e.g., POMDP models involving other healthcare problems such as liver transplantation (Sandıkc ¸ı et al, 2013) and transportation problems such as semi-autonomous driving (Wray and Zilberstein, 2015a)-may be used to further showcase the benefits of distributed implementations.…”

Section: Discussionmentioning

confidence: 99%

“…In addition, other real world problems may be solved using our methodologies to demonstrate generalizability of our approach. For example, models involving other healthcare problems such as liver transplantation (Sandıkc ¸ı et al, 2013) and transportation problems such as semi-autonomous driving (Wray and Zilberstein, 2015a) may be investigated to showcase the benefits of distributed implementations. Lastly, distributed and parallel implementations can be adopted for other approximation mechanisms such as point-based value iteration algorithm and Monte Carlo tree search algorithm.…”

Section: Challenges Limitations and Future Workmentioning

confidence: 99%

Novel Solution Methods for CPOMDPs With Applications to Cancer Screening

Kavaklioglu

2024

Preprint

0

View full text Add to dashboard Cite

<p>Partially Observable Markov Decision Processes (POMDPs) is a modeling framework that allow decision makers to make decisions in uncertain environments. This framework is particularly suited for sequential decision making problems for which the uncertainties are revealed in stages. Constrained POMDPs (CPOMDPs) extend this framework by also accounting for various restrictions in the environment. This PhD thesis focuses on POMDPs, CPOMDPs and its extensions, proposes novel solution methods and studies applications of POMDPs in disease screening. First, we focus on developing efficient versions of the well-known algorithms for POMDPs. POMDPs are notoriously difficult to solve optimally, and exact solution algorithms typically fail to solve the instances with more than a few core states. In this regard, we consider scalable algorithms solve the POMDP problems. Specifically, we propose distributed and parallel versions of the exact solution methods such as Monahan's algorithm and the incremental pruning algorithm as well as the heuristics such as Lovejoy's upper bound and lower bound methods. Next, we study the CPOMDPs and propose linear programming-based solution methods as a flexible approach for those. We perform a detailed numerical study with five different problem instances to showcase the viability of the proposed solution approach for CPOMDPs. Lastly, we propose multi-objective CPOMDP models for the breast cancer screening problem. These CPOMDP models extend the previously proposed POMDP models for breast cancer screening by considering different screening modalities and simultaneously optimizing for expected total quality-adjusted life years maximization and life-time risk minimization objectives.</p>

show abstract

“…In addition, while we provide results with a practical problem (i.e., BCS), experimenting with other practical problems and datasets -e.g., POMDP models involving other healthcare problems such as liver transplantation (Sandıkc ¸ı et al, 2013) and transportation problems such as semi-autonomous driving (Wray and Zilberstein, 2015a)-may be used to further showcase the benefits of distributed implementations.…”

Section: Discussionmentioning

confidence: 99%

“…In addition, other real world problems may be solved using our methodologies to demonstrate generalizability of our approach. For example, models involving other healthcare problems such as liver transplantation (Sandıkc ¸ı et al, 2013) and transportation problems such as semi-autonomous driving (Wray and Zilberstein, 2015a) may be investigated to showcase the benefits of distributed implementations. Lastly, distributed and parallel implementations can be adopted for other approximation mechanisms such as point-based value iteration algorithm and Monte Carlo tree search algorithm.…”

Section: Challenges Limitations and Future Workmentioning

confidence: 99%

Novel Solution Methods for CPOMDPs With Applications to Cancer Screening

Kavaklioglu

2024

Preprint

0

View full text Add to dashboard Cite

<p>Partially Observable Markov Decision Processes (POMDPs) is a modeling framework that allow decision makers to make decisions in uncertain environments. This framework is particularly suited for sequential decision making problems for which the uncertainties are revealed in stages. Constrained POMDPs (CPOMDPs) extend this framework by also accounting for various restrictions in the environment. This PhD thesis focuses on POMDPs, CPOMDPs and its extensions, proposes novel solution methods and studies applications of POMDPs in disease screening. First, we focus on developing efficient versions of the well-known algorithms for POMDPs. POMDPs are notoriously difficult to solve optimally, and exact solution algorithms typically fail to solve the instances with more than a few core states. In this regard, we consider scalable algorithms solve the POMDP problems. Specifically, we propose distributed and parallel versions of the exact solution methods such as Monahan's algorithm and the incremental pruning algorithm as well as the heuristics such as Lovejoy's upper bound and lower bound methods. Next, we study the CPOMDPs and propose linear programming-based solution methods as a flexible approach for those. We perform a detailed numerical study with five different problem instances to showcase the viability of the proposed solution approach for CPOMDPs. Lastly, we propose multi-objective CPOMDP models for the breast cancer screening problem. These CPOMDP models extend the previously proposed POMDP models for breast cancer screening by considering different screening modalities and simultaneously optimizing for expected total quality-adjusted life years maximization and life-time risk minimization objectives.</p>

show abstract

“…GPU-based parallel implementations that use the Monte-Carlo Value Iteration algorithm to solve the continuous-state POMDPs have been proposed [31,32]. Approximate point-based methods [33][34][35] have also been implemented with GPUs to solve POMDPs [36]. Despite the available methods, solving MOMDPs and POMDPs remains computationally demanding and prohibitive for large-scale problems, especially for path planning of autonomous marine vehicles.…”

Section: Prior Workmentioning

confidence: 99%

GPU-Accelerated Multi-Objective Optimal Planning in Stochastic Dynamic Environments

Chowdhury

¹

,

Navsalkar

²

,

Subramani

³

2022

JMSE

2

0

View full text Add to dashboard Cite

The importance of autonomous marine vehicles is increasing in a wide range of ocean science and engineering applications. Multi-objective optimization, where trade-offs between multiple conflicting objectives are achieved (such as minimizing expected mission time, energy consumption, and environmental energy harvesting), is crucial for planning optimal routes in stochastic dynamic ocean environments. We develop a multi-objective path planner in stochastic dynamic flows by further developing and improving our recently developed end-to-end GPU-accelerated single-objective Markov Decision Process path planner. MDPs with scalarized rewards for multiple objectives are formulated and solved in idealized stochastic dynamic ocean environments with dynamic obstacles. Three simulated mission scenarios are completed to elucidate our approach and capabilities: (i) an agent moving from a start to target by minimizing travel time and net-energy consumption when harvesting solar energy in an uncertain flow; (ii) an agent moving from a start to target by minimizing travel time and-energy consumption with uncertainties in obstacle initial positions; (iii) an agent attempting to cross a shipping channel while avoiding multiple fast moving ships in an uncertain flow. Optimal operating curves are computed in a fraction of the time that would be required for existing solvers and algorithms. Crucially, our solution can serve as the benchmark for other approximate AI algorithms such as Reinforcement Learning and help improve explainability of those models.

show abstract

A POMDP Formulation of Proactive Learning

Cited by 4 publications

References 17 publications

Novel Solution Methods for CPOMDPs With Applications to Cancer Screening

Novel Solution Methods for CPOMDPs With Applications to Cancer Screening

Novel Solution Methods for CPOMDPs With Applications to Cancer Screening

GPU-Accelerated Multi-Objective Optimal Planning in Stochastic Dynamic Environments

Contact Info

Product

Resources

About