Reinforcement Learning Based Temporal Logic Control with Maximum Probabilistic Satisfaction

Cai, Mingyu; Xiao, Shaoping; Li, Baoluo; Li, Zhiliang; Kan, Zhen

doi:10.1109/icra48506.2021.9561903

Cited by 22 publications

(13 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…6). Unlike local optimization [24,25,26], learningbased [28,29], or CBF-based [30] approaches, our proposed approach provides completeness guarantees under modest assumptions.…”

Section: Discussionmentioning

confidence: 99%

“…Finally, we acknowledge the recent trend of attempting to avoid the NP-hardness of temporal logic motion planning altogether by providing approximate solutions via non-convex optimization [24,25,26,27], learning [28,29], or control barrier functions [30,31]. While such approaches can be extremely efficient and may be practical for some applications, they offer limited or no completeness guarantees and rarely scale to very complex specifications like that shown in Fig.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Temporal Logic Motion Planning with Convex Optimization via Graphs of Convex Sets

Kurtz¹,

Lin²

2023

Preprint

View full text Add to dashboard Cite

Temporal logic is a concise way of specifying complex tasks. But motion planning to achieve temporal logic specifications is difficult, and existing methods struggle to scale to complex specifications and high-dimensional system dynamics. In this paper, we cast Linear Temporal Logic (LTL) motion planning as a shortest path problem in a Graph of Convex Sets (GCS) and solve it with convex optimization. This approach brings together the best of modern optimization-based temporal logic planners and older automata-theoretic methods, addressing the limitations of each: paths are represented with continuous Bezier curves, avoiding clipping and pass-through; computational complexity is polynomial (not exponential) in the number of sample points; global optimality can be certified; soundness and completeness are guaranteed under mild assumptions; and most importantly, the method scales to complex specifications and high-dimensional systems, including a 30-DoF humanoid. Open-source code is available at https://github.com/vincekurtz/ltl_gcs.

show abstract

“…6). Unlike local optimization [24,25,26], learningbased [28,29], or CBF-based [30] approaches, our proposed approach provides completeness guarantees under modest assumptions.…”

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Temporal Logic Motion Planning with Convex Optimization via Graphs of Convex Sets

Kurtz¹,

Lin²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, some research works employed LTL formulas to specify the instructions for a control agent to learn optimal strategies, i.e., optimal policies. Specifically, many works [11][12][13][14][15] designed automaton-based rewards so that model-free RL agents could find the optimal policies satisfying LTL specifications with probabilistic guarantees. However, none of them addresses the critical safety issues during training.…”

Section: Safe Reinforcement Learning Under Temporal Logic With Reward...mentioning

confidence: 99%

Safe reinforcement learning under temporal logic with reward design and quantum action selection

Cai

Xiao

et al. 2023

Sci Rep

Self Cite

View full text Add to dashboard Cite

This paper proposes an advanced Reinforcement Learning (RL) method, incorporating reward-shaping, safety value functions, and a quantum action selection algorithm. The method is model-free and can synthesize a finite policy that maximizes the probability of satisfying a complex task. Although RL is a promising approach, it suffers from unsafe traps and sparse rewards and becomes impractical when applied to real-world problems. To improve safety during training, we introduce a concept of safety values, which results in a model-based adaptive scenario due to online updates of transition probabilities. On the other hand, a high-level complex task is usually formulated via formal languages, including Linear Temporal Logic (LTL). Another novelty of this work is using an Embedded Limit-Deterministic Generalized Büchi Automaton (E-LDGBA) to represent an LTL formula. The obtained deterministic policy can generalize the tasks over infinite and finite horizons. We design an automaton-based reward, and the theoretical analysis shows that an agent can accomplish task specifications with the maximum probability by following the optimal policy. Furthermore, a reward shaping process is developed to avoid sparse rewards and enforce the RL convergence while keeping the optimal policies invariant. In addition, inspired by quantum computing, we propose a quantum action selection algorithm to replace the existing $$\varepsilon$$ ε -greedy algorithm for the balance of exploration and exploitation during learning. Simulations demonstrate how the proposed framework can achieve good performance by dramatically reducing the times to visit unsafe states while converging optimal policies.

show abstract

“…Specifications have been shown to be effective at directing RL agents to learn desired policies. Typically within the literature, a discrete robot system is abstracted as a discrete Markov Decision Process (MDP) model and composed with an automata representing the desired LTL formula to create a product automaton for learning or planning [5]- [8]. This approach has been extended to continuous systems [9], [10], where the LTL formulas are only defined over finite horizons.…”

Section: Introductionmentioning

confidence: 99%

Learning Minimally-Violating Continuous Control for Infeasible Linear Temporal Logic Specifications

Cai¹,

Mann²,

Serlin³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper explores continuous-time control synthesis for target-driven navigation to satisfy complex highlevel tasks expressed as linear temporal logic (LTL). We propose a model-free framework using deep reinforcement learning (DRL) where the underlying dynamic system is unknown (an opaque box). Unlike prior work, this paper considers scenarios where the given LTL specification might be infeasible and therefore cannot be accomplished globally. Instead of modifying the given LTL formula, we provide a general DRL-based approach to satisfy it with minimal violation. To do this, we transform a previously multi-objective DRL problem, which requires simultaneous automata satisfaction and minimum violation cost, into a single objective. By guiding the DRL agent with a sampling-based path planning algorithm for the potentially infeasible LTL task, the proposed approach mitigates the myopic tendencies of DRL, which are often an issue when learning general LTL tasks that can have long or infinite horizons. This is achieved by decomposing an infeasible LTL formula into several reach-avoid sub-tasks with shorter horizons, which can be trained in a modular DRL architecture. Furthermore, we overcome the challenge of the exploration process for DRL in complex and cluttered environments by using path planners to design rewards that are dense in the configuration space. The benefits of the presented approach are demonstrated through testing on various complex nonlinear systems and compared with state-of-the-art baselines. The Video demonstration can be found on YouTube Channel:https://youtu.be/DqesqBsja9k.

show abstract

Reinforcement Learning Based Temporal Logic Control with Maximum Probabilistic Satisfaction

Cited by 22 publications

References 19 publications

Temporal Logic Motion Planning with Convex Optimization via Graphs of Convex Sets

Temporal Logic Motion Planning with Convex Optimization via Graphs of Convex Sets

Safe reinforcement learning under temporal logic with reward design and quantum action selection

Learning Minimally-Violating Continuous Control for Infeasible Linear Temporal Logic Specifications

Contact Info

Product

Resources

About