Robust Partially Observable Markov Decision Processes

Rasouli, Mohammad; Saghafian, Soroush

doi:10.2139/ssrn.3195310

Cited by 12 publications

(11 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This lemma unveils the connection between POMDP and SA-MDP: SA-MDP can be seen as a version of "robust" POMDP where the policy needs to be robust under a set of observational processes (adversaries). SA-MDP is different from robust POMDP (RPOMDP) (Osogami, 2015;Rasouli & Saghafian, 2018), which optimizes for the worst case environment transitions.…”

Section: Finding the Optimal Policy Under A Fixed Adversarymentioning

confidence: 99%

Robust Reinforcement Learning on State Observations with Learned Optimal Adversary

Zhang

Chen

Boning

et al. 2021

Preprint

View full text Add to dashboard Cite

We study the robustness of reinforcement learning (RL) with adversarially perturbed state observations, which aligns with the setting of many adversarial attacks to deep reinforcement learning (DRL) and is also important for rolling out real-world RL agent under unpredictable sensing noise. With a fixed agent policy, we demonstrate that an optimal adversary to perturb state observations can be found, which is guaranteed to obtain the worst case agent reward. For DRL settings, this leads to a novel empirical adversarial attack to RL agents via a learned adversary that is much stronger than previous ones. To enhance the robustness of an agent, we propose a framework of alternating training with learned adversaries (ATLA), which trains an adversary online together with the agent using policy gradient following the optimal adversarial attack framework. Additionally, inspired by the analysis of state-adversarial Markov decision process (SA-MDP), we show that past states and actions (history) can be useful for learning a robust agent, and we empirically find a LSTM based policy can be more robust under adversaries. Empirical evaluations on a few continuous control environments show that ATLA achieves state-of-the-art performance under strong adversaries. Our code is available at https://github.com/huanzhang12/ATLA_robust_RL.

show abstract

Section: Finding the Optimal Policy Under A Fixed Adversarymentioning

confidence: 99%

Robust Reinforcement Learning on State Observations with Learned Optimal Adversary

Zhang

Chen

Boning

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The ability to solve such planning and multiagent control problems under realistic real-world uncertainty highlights the fact that undecidability of optimal policies need not prevent useful (although presumably not optimal) plans and policies from being devised-at least as long as feasible solutions can be generated and improved fairly easily to obtain good (or even, with enough iterations, approximately optimal) solutions. This is the case for POMDPs with discounted reward criteria, as well as for certain extensions of POMDPs to robust optimization settings, where model parameters are known only to lie within a specified uncertainty set, and worst-case expected cumulative reward is to be maximized (Osogami, 2015;Rasouli & Saghafian, 2018). Although computational complexity remains a formidable challenge for large POMDPs, state-of-the-art solvers use a combination of ideas (including random sampling of search trees, discretization of value functions and updates ("point-based" value iteration), together with dynamic programming and linear programming techniques) to provide solutions whose quality improves with available computational budget (Shani, Pineau, & Kaplow, 2013;Smith & Simmons, 2005).…”

Section: Multiagent Team Controlmentioning

confidence: 99%

Answerable and Unanswerable Questions in Risk Analysis with Open‐World Novelty

Cox

2020

Risk Analysis

View full text Add to dashboard Cite

Decision analysis and risk analysis have grown up around a set of organizing questions: what might go wrong, how likely is it to do so, how bad might the consequences be, what should be done to maximize expected utility and minimize expected loss or regret, and how large are the remaining risks? In probabilistic causal models capable of representing unpredictable and novel events, probabilities for what will happen, and even what is possible, cannot necessarily be determined in advance. Standard decision and risk analysis questions become inherently unanswerable ("undecidable") for realistically complex causal systems with "open-world" uncertainties about what exists, what can happen, what other agents know, and how they will act. Recent artificial intelligence (AI) techniques enable agents (e.g., robots, drone swarms, and automatic controllers) to learn, plan, and act effectively despite open-world uncertainties in a host of practical applications, from robotics and autonomous vehicles to industrial engineering, transportation and logistics automation, and industrial process control. This article offers an AI/machine learning perspective on recent ideas for making decision and risk analysis (even) more useful. It reviews undecidability results and recent principles and methods for enabling intelligent agents to learn what works and how to complete useful tasks, adjust plans as needed, and achieve multiple goals safely and reasonably efficiently when possible, despite open-world uncertainties and unpredictable events. In the near future, these principles could contribute to the formulation and effective implementation of more effective plans and policies in business, regulation, and public policy, as well as in engineering, disaster management, and military and civil defense operations. They can extend traditional decision and risk analysis to deal more successfully with open-world novelty and unpredictable events in large-scale real-world planning, policymaking, and risk management.

show abstract

“…Meanwhile, Rasouli and Saghafian (2018) consider a general setting of robust POMDP, where the DM may not be able to obtain the exact information of the nature's choice of decision. In this case, the sufficient statistic is no longer a single belief state, but a collection of belief states, and the expected reward up to the current time must be taken into account to realize a policy that is robust in terms of the entire cumulative expected reward.…”

Section: Robust and Distributionally Robust Mdpmentioning

confidence: 99%

“…In this case, the sufficient statistic is no longer a single belief state, but a collection of belief states, and the expected reward up to the current time must be taken into account to realize a policy that is robust in terms of the entire cumulative expected reward. Rasouli and Saghafian (2018) derives an exact algorithm for the case where the uncertainty set is discrete. Here we note that robust POMDP for continuous support of uncertainty sets is computationally challenging even in a very simple setting (Nakao et al, 2019).…”

Section: Robust and Distributionally Robust Mdpmentioning

confidence: 99%

Distributionally Robust Partially Observable Markov Decision Process with Moment-based Ambiguity

Nakao¹,

Jiang²,

Shen

2019

Preprint

View full text Add to dashboard Cite

We consider a distributionally robust (DR) formulation of partially observable Markov decision process (POMDP), where the transition probabilities and observation probabilities are random and unknown, only revealed at the end of every time step. We construct the ambiguity set of the joint distribution of the two types of probabilities using moment information bounded via conic constraints and show that the value function of DR-POMDP is convex with respect to the belief state. We propose a heuristic search value iteration method to solve DR-POMDP, which finds lower and upper bounds of the optimal value function. Computational analysis is conducted to compare DR-POMDP with the standard POMDP using random instances of dynamic machine repair and a ROCKSAMPLE benchmark.

show abstract

Robust Partially Observable Markov Decision Processes

Cited by 12 publications

References 29 publications

Robust Reinforcement Learning on State Observations with Learned Optimal Adversary

Robust Reinforcement Learning on State Observations with Learned Optimal Adversary

Answerable and Unanswerable Questions in Risk Analysis with Open‐World Novelty

Distributionally Robust Partially Observable Markov Decision Process with Moment-based Ambiguity

Contact Info

Product

Resources

About