Certified Reinforcement Learning with Logic Guidance

Abstract

This paper proposes the first model-free Reinforcement Learning (RL) framework to synthesise policies for unknown, and continuous-state Markov Decision Processes (MDPs), such that a given linear temporal property is satisfied. We convert the given property into a Limit Deterministic Büchi Automaton (LDBA), namely a finite-state machine expressing the property. Exploiting the structure of the LDBA, we shape a synchronous reward function on-the-fly, so that an RL algorithm can synthesise a policy resulting in traces that probabilistically satisfy the linear temporal property. This probability (certificate) is also calculated in parallel with learning when the MDP state space is finite: as such, the RL algorithm produces a policy that is certified with respect to the property. Under the assumption of finite state space, theoretical guarantees are provided on the convergence of the RL algorithm to optimal policies, maximising the above probability. We also show that our method produces "best available" control policies when the logical property cannot be satisfied. In the general case of a continuous state space, we propose a neural network architecture for RL and we empirically show that the algorithm finds satisfying policies, if there exist such policies. The performance of the proposed framework is evaluated via a set of numerical examples and benchmarks, where we observe an improvement of one order of magnitude in the number of iterations required for the policy synthesis, compared to existing approaches whenever available. IntroductionMarkov Decision Processes (MDPs) are a family of stochastic processes adopted in automatic control, computer science, economics, and biology inter alia, for modelling sequential decision-making problems [85]. By choosing relevant actions over states, a decision maker (or an agent) can move probabilistically

Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logic

“…Such property is the innovation of EP-MDP that encourages all accepting sets to be visited in each round. Since the action is selected randomly, there is a non-zero probability that x P violates the LTL task (line [14][15][16].…”

“…The upper bound m = max M M < P tr π P R rec π and m ≥ 0, where P is a block matrix whose nonzero entries are derived similarly to p in (14). The utility U tr π (x R ) can be lower bounded from ( 14) and (15) as…”

“…We assign each DDPG an individual replay buffer B qi and a random process noise N qi . The corresponding weights of modular networks, i.e., Q qi x, u P θ Q q i and π qi (x |θ uq i ), are also updated at each iteration (line [15][16][17][18][19][20]. All neural networks are trained using their own replay buffer, which is a finite-sized cache that stores transitions sampled from exploring the environment.…”

“…However, scalability is a pressing issue for applying model-based approaches due to the need to store the learned model. On the other hand, by relaxing the need to construct an MDP model, model-free RL is recently adopted where appropriate reward shaping schemes are proposed [12][13][14][15][16][17][18][19][20].…”

Jump Operator Planning: Goal-Conditioned Policy Ensembles and Zero-Shot Transfer

“…Research regarding goal-conditioned polices was introduced both in the context of RL [11], and in the context of model-based non-stationary control [12]. Ordered-goal tasks have also been recently proposed in RL for training agents with symbolic and temporal rules [13][14][15][16][17][18][19][20]. Policy-concatenation (stitching) has also been proposed as a means of skill-learning in the Options framework [21][22][23], as well as Hierarchical Control [24,12], and other schemes for concatenating polices such as policy sketches (using modular polices and task-specific reward functions) [25], or compositional plan vectors [26].…”