Strategic planning under uncertainties via constrained Markov Decision Processes

Abstract

In this paper, we propose a hierarchical mission planner where the state of the world and of the mission are abstracted into corresponding states of a Markov Decision Process (MDP). Transitions in the MDP represent abstract motion actions that are planned by a lower level probabilistic planner. The cost structure of the MDP is multi-dimensional: each state-action pair is annotated with a vector of metrics such as time and resource requirements. A mission specification is divided into three parts: a temporal logic formula defined over state propositions, the choice of the primary cost, and constraints on the remaining secondary costs. The planning problem is formulated as finding the optimal policy of a Constrained Markov Decision Process with above mission specification. The resulting planning system is tested in a mission where an agent is tasked with a complex mission in a urban hostile environment.

Rapid multirobot deployment with time constraints

“…Markov decision processes have been extensively used in robotics and any list is necessarily incomplete. However, the use of Constrained Markov Decision Processes [2] in robotics is much more limited, in particular when it comes to deployment-like problems [12].…”

Risk-aware path planning using hirerachical constrained Markov Decision Processes

“…Constrained Markov Decision Processes have been used in numerous engineering domains [2], but their application in the area of robotics and automation is limited [9]. To the best of our knowledge, the use of hierarchical methods based on CMDPs for risk aware path planning appears to be an unexplored area, and in general hierarchical CMDPs have not been extensively investigated.…”

Multi-objective planning with multiple high level task specifications

“…When comparing sc-LTL formulas with full LTL formulas, the reader will note that the set of temporal operators misses the always operator. As pointed out in [6], since robots operate over missions with finite duration, the always operator is mostly irrelevant when specifying mission objectives and can therefore be omitted. sc-LTL properties are verified in a finite amount of time, as opposed to safety properties that are violated in a finite amount of time.…”

“…LTL formulas are used to specify properties of reactive systems, and the terms formula and property are then used interchangeably in the following. We consider a subset of LTL formulas leading to the so called syntactically co-safe LTL (sc-LTL) properties [6]. Starting from a set of atomic propositions Π, a sc-LTL formula is built using the standard operators and (∧), or (∨), not (¬), and the temporal operators eventually (♦), next ( ), and until (U ).…”

“…Most of these approaches optimize over one objective function only, although some consider multiple objectives [5], [6], [10], [11] through the use of CMDPs. LTL is being increasingly used to formulate task specifications for robotics and automation [16] and there have been some efforts to solve MDPs with LTL constraints.…”