Approximate Dynamic Programming with Probabilistic Temporal Logic Constraints

Abstract

In this paper, we develop approximate dynamic programming methods for stochastic systems modeled as Markov Decision Processes, given both soft performance criteria and hard constraints in a class of probabilistic temporal logic called Probabilistic Computation Tree Logic (PCTL). Our approach consists of two steps: First, we show how to transform a class of PCTL formulas into chance constraints that can be enforced during planning in stochastic systems. Second, by integrating randomized optimization and entropy-regulated dynamic programming, we devise a novel trajectory samplingbased approximate value iteration method to iteratively solve for an upper bound on the value function while ensuring the constraints that PCTL specifications are satisfied. Particularly, we show that by the on-policy sampling of the trajectories, a tight bound can be achieved between the upper bound given by the approximation and the true value function. The correctness and efficiency of the method are demonstrated using robotic motion planning examples.

Topological Approximate Dynamic Programming under Temporal Logic Constraints

“…However, if we directly solve for approximately optimal policies in the product MDP using the method in Section III-B, as the reward is sparse, it becomes a rare event to sample a path satisfying the specification. As a consequence, the estimate of the gradient in [14] has a high variance with finite samples. To address this problem, we develop Topological Approximate Dynamic Programming (TADP) that leverages the structure property in the task automaton to improve the convergence due to sparse and temporally extended rewards with LTL specifications.…”

“…In this section, we present a model-free ADP method for value iteration. The method has been introduced in our previous work [14] and will be briefly reviewed here for completeness.…”

“…The policy π(·; θ) : S → ∆(A) is computed from V (·; θ) using (2). It is shown in [14] that a value function V (·; θ) satisfying the constraint in (8) is an upper bound on the optimal value function. The objective is to minimize this upper bound to approximate the optimal value function.…”

Policy Synthesis for Metric Interval Temporal Logic with Probabilistic Distributions

“…With abstraction methods such as region automaton [20], we can potentially reduce the size of the approximated finite-state model. We are also considering incorporating approximate dynamic programming [1] for large-scale MDPs to handle the issue of scalability.…”

“…The task specifies the assumption about the probabilistic external events and the desired agent's behavior with timing constraints related to the occurrence of the external events. Such problem is widely encountered in robotics [1], [2], and other cyber-physical systems [3].…”