Loss bounds for uncertain transition probabilities in Markov decision processes

Mastin, Andrew; Jaillet, Patrick

doi:10.1109/cdc.2012.6426504

Cited by 13 publications

(8 citation statements)

References 17 publications

(15 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Instead of adopting existing approaches in non-linearity measure, SNM adopts the approach commonly used for sensitivity analysis [21,22] of Markov Decision Processes (MDP) -a special class of POMDP where uncertainty is only in the effect of performing actions. It is based on the statistical distance measure between the true transition dynamics and its perturbed versions.…”

Section: Related Workmentioning

confidence: 99%

Linearization in Motion Planning under Uncertainty

Hoerger

Kurniawati

Bandyopadhyay

et al. 2020

Springer Proceedings in Advanced Robotics

View full text Add to dashboard Cite

Motion planning under uncertainty is essential to autonomous robots. Over the past decade, the scalability of such planners have advanced substantially. Despite these advances, the problem remains difficult for systems with non-linear dynamics. Most successful methods for planning perform forward search that relies heavily on a large number of simulation runs. Each simulation run generally requires more costly integration for systems with non-linear dynamics. Therefore, for such problems, the entire planning process remains relatively slow. Not surprisingly, linearization-based methods for planning under uncertainty have been proposed. However, it is not clear how linearization affects the quality of the generated motion strategy, and more importantly where to and where not to use such a simplification. This paper presents our preliminary work towards answering such questions. In particular, we propose a measure, called Statistical-distance-based Non-linearity Measure (SNM), to identify where linearization can and where it should not be performed. The measure is based on the distance between the distributions that represent the original motion-sensing models and their linearized version. We show that when the planning problem is framed as the Partially Observable Markov Decision Process (POMDP), the difference between the value of the optimal strategy generated if we plan using the original model and if we plan using the linearized model, can be upper bounded by a function linear in SNM. We test the applicability of this measure in simulation via two venues. First, we compare SNM with a negentropy-based Measure of Non-Gaussianity (MoNG) -a measure that has recently been shown to be a suitable measure of non-linearity for stochastic systems [1]. We compare their performance in measuring the difference between a general POMDP solver [2] that computes motion strategies using the original model and a solver that uses the linearized model (adapted from [3]) on various scenarios. Our results indicate that SNM is more suitable in taking into account the effect that obstacles have on the effectiveness of linearization. In the second set of tests, we use a local estimate of SNM to develop a simple on-line planner that switches between using the original and the linearized model. Simulation results on a car-like robot with second order dynamics and a 4-DOFs and 6-DOFs manipulator with torque control indicate that our simple planner appropriately decides if and when linearization should be used.

show abstract

Section: Related Workmentioning

confidence: 99%

Linearization in Motion Planning under Uncertainty

Hoerger

Kurniawati

Bandyopadhyay

et al. 2020

Springer Proceedings in Advanced Robotics

View full text Add to dashboard Cite

show abstract

“…We now show that the proposed algorithm converges to the optimal long-term average cost. This analysis follows from the favorable redundancy properties of the CTW probability estimates, and some results from the theory of exact dynamic programming with estimated transition probabilities [53]. The basic idea is to define criteria for transition probability and cost-to-go estimates for which acting according to our estimated values will be equivalent to acting on P and J respectively.…”

Section: A Asymptotic Analysismentioning

confidence: 99%

“…The existence of ε follows immediately from the fact that the dynamic programming operator is a contraction mapping [49, Proposition 4.1] and the finiteness of the state-action space. The optimality of acting on these estimates follows from the assumption of bounded cost per stage [53,Theorem 2].…”

Section: A Asymptotic Analysismentioning

confidence: 99%

Universal Learning Waveform Selection Strategies for Adaptive Target Tracking

Thornton¹,

Buehrer²,

Dhillon³

et al. 2022

Preprint

View full text Add to dashboard Cite

Online selection of optimal waveforms for target tracking with active sensors has long been a problem of interest. Many conventional solutions utilize an estimation-theoretic interpretation, in which a waveform-specific Cram\'{e}r-Rao lower bound on measurement error is used to select the optimal waveform for each tracking step. However, this approach is only valid in the high SNR regime, and requires a rather restrictive set of assumptions regarding the target motion and measurement models. Further, due to computational concerns, many traditional approaches are limited to near-term, or \emph{myopic}, optimization, even though radar scenes exhibit strong temporal correlation. More recently, reinforcement learning has been proposed for waveform selection, in which the problem is framed as a Markov decision process (MDP), allowing for long-term planning. However, a major limitation of reinforcement learning is that the memory length of the underlying Markov process is often unknown for realistic target and channel dynamics, and a more general framework is desirable. This work develops a universal sequential waveform selection scheme which asymptotically achieves Bellman optimality in any radar scene which can be modeled as a th order Markov process for a finite, but unknown, integer U. Our approach is based on well-established tools from the field of universal source coding, where a stationary source is parsed into variable length phrases in order to build a context-tree, which is used as a probabalistic model for the scene's behavior. We show that an algorithm based on a multi-alphabet version of the Context-Tree Weighting (CTW) method can be used to optimally solve a broad class of waveform-agile tracking problems while making minimal assumptions about the environment's behavior.

show abstract

“…SNM is designed to address these issues. Instead of building upon existing non-linearity measures, SNM adopts approaches commonly used for sensitivity analysis [23], [24] of Markov Decision Processes (MDP) -a special class of POMDP where the observation model is perfect, and therefore the system is fully observable. These approaches use statistical distance measures between the original transition dynamics and their perturbed versions.…”

Section: B Related Work On Non-linearity Measuresmentioning

confidence: 99%

Non-Linearity Measure for POMDP-based Motion Planning

Hoerger¹,

Kurniawati²,

Elfes³

2020

Preprint

View full text Add to dashboard Cite

Motion planning under uncertainty is essential for reliable robot operation. Despite substantial advances over the past decade, the problem remains difficult for systems with complex dynamics. Most state-of-the-art methods perform search that relies on a large number of forward simulations. For systems with complex dynamics, this generally require costly numerical integrations which significantly slows down the planning process. Linearization-based methods have been proposed that can alleviate the above problem. However, it is not clear how linearization affects the quality of the generated motion strategy, and when such simplifications are admissible. We propose a non-linearity measure, called Statistical-distance-based Non-linearity Measure (SNM), that can identify where linearization is beneficial and where it should be avoided. We show that when the problem is framed as the Partially Observable Markov Decision Process, the value difference between the optimal strategy for the original model and the linearized model can be upper bounded by a function linear in SNM. Comparisons with an existing measure on various scenarios indicate that SNM is more suitable in estimating the effectiveness of linearization-based solvers. To test the applicability of SNM in motion planning, we propose a simple on-line planner that uses SNM as a heuristic to switch between a general and a linearization-based solver. Results on a car-like robot with second order dynamics and 4-DOFs and 7-DOFs torque-controlled manipulators indicate that SNM can appropriately decide if and when a linearization-based solver should be used.

show abstract

Loss bounds for uncertain transition probabilities in Markov decision processes

Cited by 13 publications

References 17 publications

Linearization in Motion Planning under Uncertainty

Linearization in Motion Planning under Uncertainty

Universal Learning Waveform Selection Strategies for Adaptive Target Tracking

Non-Linearity Measure for POMDP-based Motion Planning

Contact Info

Product

Resources

About