Induction of Subgoal Automata for Reinforcement Learning

Furelos-Blanco, Daniel; Law, Mark; Russo, Alessandra; Broda, Krysia; Jönsson, Anders

doi:10.1609/aaai.v34i04.5802

Cited by 20 publications

(16 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where γ is the MDP's discount factor and Φ : S → R is a real-valued function. The automaton structure can be exploited by defining F : (U \ {u A , u R }) × U → R in terms of the automaton states instead (Camacho et al, 2019;Furelos-Blanco et al, 2020):…”

Section: Option Modeling Given a Subgoal Automatonmentioning

confidence: 99%

“…In our previous work (Furelos-Blanco et al, 2020), we introduced a method for breaking symmetries in acyclic subgoal automata, which consists in:…”

Section: Symmetry Breakingmentioning

confidence: 99%

“…. , u 3 and apply the symmetry breaking rule in (Furelos-Blanco et al, 2020), the learned automaton for OfficeWorld's VisitABCD task will always be the one showed in Figure 3a. On the other hand, the automaton states u 1 and u 2 in Figure 3b can still be switched since there is no trace that traverses both of them.…”

Section: Symmetry Breakingmentioning

confidence: 99%

“…In particular, Toro Icarte et al (2018) recently proposed reward machines (RMs), which are automata that play the role of reward functions while revealing the task's structure to the RL agent. This approach has drawn attention from the community and several works have recently attempted to learn RMs (Toro Icarte et al, 2019;Xu et al, 2020) or similar kinds of automata (Furelos-Blanco et al, 2020;Gaon & Brafman, 2020).…”

Section: Introductionmentioning

confidence: 99%

“…The description of our approach previously appeared in a conference paper (Furelos-Blanco et al, 2020). Compared to the conference version, the present paper includes the following novel material:…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Induction and Exploitation of Subgoal Automata for Reinforcement Learning

Furelos-Blanco

Law

Jönsson

et al. 2021

jair

Self Cite

View full text Add to dashboard Cite

In this paper we present ISA, an approach for learning and exploiting subgoals in episodic reinforcement learning (RL) tasks. ISA interleaves reinforcement learning with the induction of a subgoal automaton, an automaton whose edges are labeled by the task’s subgoals expressed as propositional logic formulas over a set of high-level events. A subgoal automaton also consists of two special states: a state indicating the successful completion of the task, and a state indicating that the task has finished without succeeding. A state-of-the-art inductive logic programming system is used to learn a subgoal automaton that covers the traces of high-level events observed by the RL agent. When the currently exploited automaton does not correctly recognize a trace, the automaton learner induces a new automaton that covers that trace. The interleaving process guarantees the induction of automata with the minimum number of states, and applies a symmetry breaking mechanism to shrink the search space whilst remaining complete. We evaluate ISA in several gridworld and continuous state space problems using different RL algorithms that leverage the automaton structures. We provide an in-depth empirical analysis of the automaton learning performance in terms of the traces, the symmetry breaking and specific restrictions imposed on the final learnable automaton. For each class of RL problem, we show that the learned automata can be successfully exploited to learn policies that reach the goal, achieving an average reward comparable to the case where automata are not learned but handcrafted and given beforehand.

show abstract

Section: Option Modeling Given a Subgoal Automatonmentioning

confidence: 99%

“…In our previous work (Furelos-Blanco et al, 2020), we introduced a method for breaking symmetries in acyclic subgoal automata, which consists in:…”

Section: Symmetry Breakingmentioning

confidence: 99%

Section: Symmetry Breakingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Induction and Exploitation of Subgoal Automata for Reinforcement Learning

Furelos-Blanco

Law

Jönsson

et al. 2021

jair

Self Cite

View full text Add to dashboard Cite

show abstract

A survey of knowledge‐based sequential decision‐making under uncertainty

Zhang

Sridharan

2022

AI Magazine

View full text Add to dashboard Cite

Reasoning with declarative knowledge (RDK) and sequential decision-making (SDM) are two key research areas in artificial intelligence. RDK methods reason with declarative domain knowledge, including commonsense knowledge, that is either provided a priori or acquired over time, while SDM methods (probabilistic planning [PP] and reinforcement learning [RL]) seek to compute action policies that maximize the expected cumulative utility over a time horizon; both classes of methods reason in the presence of uncertainty. Despite the rich literature in these two areas, researchers have not fully explored their complementary strengths.In this paper, we survey algorithms that leverage RDK methods while making sequential decisions under uncertainty. We discuss significant developments, open problems, and directions for future work.

show abstract

Active Finite Reward Automaton Inference and Reinforcement Learning Using Queries and Counterexamples

Ojha

Neider

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Induction of Subgoal Automata for Reinforcement Learning

Cited by 20 publications

References 29 publications

Induction and Exploitation of Subgoal Automata for Reinforcement Learning

Induction and Exploitation of Subgoal Automata for Reinforcement Learning

A survey of knowledge‐based sequential decision‐making under uncertainty

Active Finite Reward Automaton Inference and Reinforcement Learning Using Queries and Counterexamples

Contact Info

Product

Resources

About