Transfer of Temporal Logic Formulas in Reinforcement Learning

Abstract

Transferring high-level knowledge from a source task to a target task is an effective way to expedite reinforcement learning (RL). For example, propositional logic and first-order logic have been used as representations of such knowledge. We study the transfer of knowledge between tasks in which the timing of the events matters. We call such tasks temporal tasks. We concretize similarity between temporal tasks through a notion of logical transferability, and develop a transfer learning approach between different yet similar temporal tasks. We first propose an inference technique to extract metric interval temporal logic (MITL) formulas in sequential disjunctive normal form from labeled trajectories collected in RL of the two tasks. If logical transferability is identified through this inference, we construct a timed automaton for each sequential conjunctive subformula of the inferred MITL formulas from both tasks. We perform RL on the extended state which includes the locations and clock valuations of the timed automata for the source task. We then establish mappings between the corresponding components (clocks, locations, etc.) of the timed automata from the two tasks, and transfer the extended Q-functions based on the established mappings. Finally, we perform RL on the extended state for the target task, starting with the transferred extended Q-functions. Our results in two case studies show, depending on how similar the source task and the target task are, that the sampling efficiency for the target task can be improved by up to one order of magnitude by performing RL in the extended state space, and further improved by up to another order of magnitude using the transferred extended Q-functions. IntroductionReinforcement learning (RL) has been successful in numerous applications. In practice though, it often requires extensive exploration of the environment to achieve satisfactory performance, especially for complex tasks with sparse rewards [1].The sampling efficiency and performance of RL can be improved if some high-level knowledge can be incorporated in the learning process [2]. Such knowledge can be also transferred from a source task to a target task if these tasks are logically similar [3]. For example, propositional logic and first-order logic have been used as representations of knowledge in the form of logical structures for transfer learning [4]. They showed that incorporating such logical similarities can expedite RL for the target task [5].The transfer of high-level knowledge can be also applied to tasks where the timing of the events matters. We call such tasks as temporal tasks.

Robotics Dexterous Grasping: The Methods Based on Point Cloud and Deep Learning

“…However, for the proposed approaches based on reinforcement learning are hard to be transferred to other tasks. Even if there are already some works on the integration of transfer learning and reinforcement learning (Tirinzoni et al, 2018 ; Ammanabrolu and Riedl, 2019 ; Gamrian and Goldberg, 2019 ; Liu Y. et al, 2019 ; Xu and Topcu, 2019 ), the work related to the robot grasping is lacking. The essence of transfer learning is the registration problem at the task level, including not only the task itself, but also its input and output.…”

Control strategies for COVID-19 epidemic with vaccination, shield immunity and quarantine: A metric temporal logic approach

“…We use metric temporal logic (MTL) formulas to specify the expected control outcomes such as "the deaths from the infection should never exceed one thousand per day within the next three months" or "the population immune from the disease should eventually exceed 200 thousand within the next 100 to 120 days". Such temporal logic formulas have been used as high-level knowledge or specifications in many applications in artificial intelligence [8], robotic control [9], power systems [10], etc.…”

Compositional Reinforcement Learning from Logical Specifications

“…There has been recent work on using specifications based on temporal logic for specifying RL tasks [2,15,26,14,35,12,34,19]. These approaches typically generate a (usually sparse) reward function from a given specification which is then used by an off-the-shelf RL algorithm to learn a policy.…”