Modular Deep Reinforcement Learning for Continuous Motion Planning With Temporal Logic

Cai, Mingyu; Hasanbeig, Mohammadhosein; Xiao, Shaoping; Abate, Alessandro; Kan, Zhen

doi:10.1109/lra.2021.3101544

Cited by 60 publications

(17 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Baseline Approaches: From the learning perspective, we refer to our distributed framework as "RRT*" or "D-RRT*", and compare it against three baselines: (i) The relaxed TLbased multi-objective rewards in [11], [31] referred to as "TL", for the single LTL task; (ii) For the goal-reaching task φ, the baseline referred to as "NED" designs the reward based on the negative Euclidean distance between the robot and destination; (iii) For a complex LTL task, instead of decomposition, this baseline directly applies the reward scheme (6) for the global trajectory τ * F = τ * pre [τ * suf ] ω referred to as "G-RRT*". From the perspective of infeasible LTL tasks, we compare with the work [30] of visiting as many accepting sets of LDGBA [33] as possible, and we empirically show the improvement over our prior work [13] that assumes the feasible cases.…”

Section: Resultsmentioning

confidence: 99%

“…The next operator is not always meaningful since it may require an immediate execution switch in the synthesized plans space [42]. Based on the application, we can either exclude the next operator as is common in related work [38], [42] or properly design practical LTL tasks [11] in continuous scenarios.…”

Section: Preliminariesmentioning

confidence: 99%

“…Furthermore, DRL-based navigation control typically uses either a sparse reward function that assigns positive rewards only when the robot reaches a destination, or leverages distance-to-goal measurements for a smoother reward. For LTL specifications, discrete goal-reaching may enable the automaton transitions, which is one requirement to receive automata-based rewards [9], [11]. However, in cluttered environments such as those depicted in Fig.…”

Section: Preliminariesmentioning

confidence: 99%

“…This approach has been extended to continuous systems [9], [10], where the LTL formulas are only defined over finite horizons. For general continuous control subject to LTL satisfaction over infinite horizons, prior works [11]- [13] proposed a modular architecture by decomposing the task into sub-tasks using automata states. All this related work designs RL rewards according to the discrete switch of an automata, which requires the RL agent to visit every region of interest to gain experience in each automata state.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Learning Minimally-Violating Continuous Control for Infeasible Linear Temporal Logic Specifications

Cai¹,

Mann²,

Serlin³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper explores continuous-time control synthesis for target-driven navigation to satisfy complex highlevel tasks expressed as linear temporal logic (LTL). We propose a model-free framework using deep reinforcement learning (DRL) where the underlying dynamic system is unknown (an opaque box). Unlike prior work, this paper considers scenarios where the given LTL specification might be infeasible and therefore cannot be accomplished globally. Instead of modifying the given LTL formula, we provide a general DRL-based approach to satisfy it with minimal violation. To do this, we transform a previously multi-objective DRL problem, which requires simultaneous automata satisfaction and minimum violation cost, into a single objective. By guiding the DRL agent with a sampling-based path planning algorithm for the potentially infeasible LTL task, the proposed approach mitigates the myopic tendencies of DRL, which are often an issue when learning general LTL tasks that can have long or infinite horizons. This is achieved by decomposing an infeasible LTL formula into several reach-avoid sub-tasks with shorter horizons, which can be trained in a modular DRL architecture. Furthermore, we overcome the challenge of the exploration process for DRL in complex and cluttered environments by using path planners to design rewards that are dense in the configuration space. The benefits of the presented approach are demonstrated through testing on various complex nonlinear systems and compared with state-of-the-art baselines. The Video demonstration can be found on YouTube Channel:https://youtu.be/DqesqBsja9k.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Preliminariesmentioning

confidence: 99%

Section: Preliminariesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Learning Minimally-Violating Continuous Control for Infeasible Linear Temporal Logic Specifications

Cai¹,

Mann²,

Serlin³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The known cart-pole experiment (Fig. 3.b) [8,25,44] has a task that is expressed by the LTL specification ♦y ∧ ♦g ∧ ¬u, namely, starting the pole in upright position, the goal is to prevent it from falling over ( ¬u, namely always not u) by moving the cart, whilst in particular alternating between the yellow (y) and green (g) regions ( ♦y ∧ ♦g), while avoiding the red (unsafe) parts of the track ( ¬u).…”

Section: Experimental Evaluationmentioning

confidence: 99%

LCRL: Certified Policy Synthesis via Logically-Constrained Reinforcement Learning

Hasanbeig

Kroening²,

Abate³

2022

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

LCRL is a software tool that implements model-free Reinforcement Learning (RL) algorithms over unknown Markov Decision Processes (MDPs), synthesising policies that satisfy a given linear temporal specification with maximal probability. LCRL leverages partially deterministic finite-state machines known as Limit Deterministic Büchi Automata (LDBA) to express a given linear temporal specification. A reward function for the RL algorithm is shaped on-the-fly, based on the structure of the LDBA. Theoretical guarantees under proper assumptions ensure the convergence of the RL algorithm to an optimal policy that maximises the satisfaction probability. We present case studies to demonstrate the applicability, ease of use, scalability and performance of LCRL . Owing to the LDBA-guided exploration and LCRL model-free architecture, we observe robust performance, which also scales well when compared to standard RL approaches (whenever applicable to LTL specifications). Full instructions on how to execute all the case studies in this paper are provided on a GitHub page that accompanies the LCRL distribution www.github.com/grockious/lcrl.

show abstract

Active Finite Reward Automaton Inference and Reinforcement Learning Using Queries and Counterexamples

Ojha

Neider

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Modular Deep Reinforcement Learning for Continuous Motion Planning With Temporal Logic

Cited by 60 publications

References 21 publications

Learning Minimally-Violating Continuous Control for Infeasible Linear Temporal Logic Specifications

Learning Minimally-Violating Continuous Control for Infeasible Linear Temporal Logic Specifications

LCRL: Certified Policy Synthesis via Logically-Constrained Reinforcement Learning

Active Finite Reward Automaton Inference and Reinforcement Learning Using Queries and Counterexamples

Contact Info

Product

Resources

About