Encoding Human Domain Knowledge to Warm Start Reinforcement Learning

Abstract

Deep reinforcement learning has been successful in a variety of tasks, such as game playing and robotic manipulation. However, attempting to learn tabula rasa disregards the logical structure of many domains as well as the wealth of readily available knowledge from domain experts that could help "warm start" the learning process. We present a novel reinforcement learning technique that allows for intelligent initialization of a neural network weights and architecture. Our approach permits the encoding domain knowledge directly into a neural decision tree, and improves upon that knowledge with policy gradient updates. We empirically validate our approach on two OpenAI Gym tasks and two modified StarCraft 2 tasks, showing that our novel architecture outperforms multilayer-perceptron and recurrent architectures. Our knowledge-based framework finds superior policies compared to imitation learning-based and prior knowledge-based approaches. Importantly, we demonstrate that our approach can be used by untrained humans to initially provide >80% increase in expected reward relative to baselines prior to training (p < 0.001), which results in a >60% increase in expected reward after policy optimization (p = 0.011).

Towards Reconciling Usability and Usefulness of Explainable AI Methodologies

“…However, it has been found that approaches that rely on visual assessment can sometimes be misleading, as they may be specific to unique data or modelling conditions, and can be highly susceptible to outlying outputs that contradict the explanation [33][34][35]. Prior work has also sought to transform uninterpretable deep networks into interpretable architectures or modalities such as decision trees [12,13,36], or bayesian rule lists [14], and generate explanations by exploiting the "white-box" nature of these architectures [37].…”

“…Explanation Format -We chose XAI modalities that can all be generated from a single methodology presented in prior work [13]. This method converts learned policies into discretized decision trees which elucidate the decision making process within an AI-agent's policy [37].…”

“…Transparency-based approaches seek to expose the internal mechanisms of an algorithm in a simpler format. Prior work has established strong baselines for transparency through gradient-based relevance methods [10,11], or presented inherently interpretable architectures wherein the explainability comes from the nature of the architecture [12][13][14]. Contrarily, post-hoc approaches provide additional context to a user Figure 1: The job of an XAI agent is to present the user with an explanation they can apply towards understanding how the AI agent works.…”

“…Decision trees have become a popular method of explaining decisions for human-AI teaming scenarios [36]. Differential decision trees have been proven to be an interpretable method of representing a policies that can be employed towards generating "white-box" explanations for users that actually represent the underlying behavior [13]. Language has also shown to be an effective means of explaining the actions of an sequential decision making agent [2,38,50].…”

Transferring Multiple Policies to Hotstart Reinforcement Learning in an Air Compressor Management Problem

“…Finally, Warm Start Reinforcement Learning (WSRL) Cheng et al (2018); Zhu & Liao (2017) aims at initializing the policy of the agent with another policy pre-trained on the same task. Domain knowledge, i.e., information about the environment known by the designers but not initially known by the agent, can be used to kickstart learning, either through imitation learning on expert demonstrations Cheng et al (2018), directly encoding it via propositional rules in the neural network architecture of the agent Silva & Gombolay (2021), or actively learning to imitate a transferred policy Wexler et al (2022).…”