Conservative Q-Improvement: Reinforcement Learning for an Interpretable Decision-Tree Policy

Roth, Aaron M.; Topin, Nicholay; Jamshidi, Pooyan; Veloso, Manuela

doi:10.48550/arxiv.1907.01180

Cited by 13 publications

(15 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several works [5,6,7,8] use greedy heuristics to induce the trees. However, this approaches have the following drawbacks:…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Evolutionary learning of interpretable decision trees

Custode,

Iacca

2020

Preprint

View full text Add to dashboard Cite

Reinforcement learning techniques achieved human-level performance in several tasks in the last decade. However, in recent years, the need for interpretability emerged: we want to be able to understand how a system works and the reasons behind its decisions. Not only we need interpretability to assess the safety of the produced systems, we also need it to extract knowledge about unknown problems. While some techniques that optimize decision trees for reinforcement learning do exist, they usually employ greedy algorithms or they do not exploit the rewards given by the environment. This means that these techniques may easily get stuck in local optima. In this work, we propose a novel approach to interpretable reinforcement learning that uses decision trees. We present a two-level optimization scheme that combines the advantages of evolutionary algorithms with the advantages of Q-learning. This way we decompose the problem into two sub-problems: the problem of finding a meaningful and useful decomposition of the state space, and the problem of associating an action to each state. We test the proposed method on three well-known reinforcement learning benchmarks, on which it results competitive with respect to the stateof-the-art in both performance and interpretability. Finally, we perform an ablation study that confirms that using the two-level optimization scheme gives a boost in performance in non-trivial environments with respect to a one-layer optimization technique.

show abstract

“…Several works [5,6,7,8] use greedy heuristics to induce the trees. However, this approaches have the following drawbacks:…”

Section: Methodsmentioning

confidence: 99%

“…In [5] the authors propose a method that predicts the gain obtained by adding a split to the tree and select the best split to grow the tree. The experimental results show that this method is more effective than the method proposed in [8] on the tested environment.…”

Section: Theorymentioning

confidence: 99%

Evolutionary learning of interpretable decision trees

Custode,

Iacca

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…The authors show that their proposition can achieve comparable performance to the original non-interpretable policy, and is amenable to verification. As an alternative approach to control the decision tree size, [200] proposes to increase its size only if the novel decision tree increases sufficiently the performance. As an improvement to work (like VIPER) using only one decision tree, In [201], a mixture of Expert Trees (MOET) is proposed.…”

Section: Decision Trees and Variantsmentioning

confidence: 99%

A Survey on Interpretable Reinforcement Learning

Glanois¹,

Weng²,

Zimmer³

et al. 2021

Preprint

View full text Add to dashboard Cite

Although deep reinforcement learning has become a promising machine learning approach for sequential decision-making problems, it is still not mature enough for high-stake domains such as autonomous driving or medical applications. In such contexts, a learned policy needs for instance to be interpretable, so that it can be inspected before any deployment (e.g., for safety and verifiability reasons). This survey provides an overview of various approaches to achieve higher interpretability in reinforcement learning (RL). To that aim, we distinguish interpretability (as a property of a model) and explainability (as a post-hoc operation, with the intervention of a proxy) and discuss them in the context of RL with an emphasis on the former notion. In particular, we argue that interpretable RL may embrace different facets: interpretable inputs, interpretable (transition/reward) models, and interpretable decision-making. Based on this scheme, we summarize and analyze recent work related to interpretable RL with an emphasis on papers published in the past 10 years. We also discuss briefly some related research areas and point to some potential promising research directions.

show abstract

“…Other explanation methods include decision tree (Bastani et al, 2018;Gupta et al, 2015;Roth et al, 2019) and structural causal MDP (Madumal et al, 2019;Waa et al, 2018). While decision tree can be represented graphically and thus aid in human understanding, a reasonably-sized tree with explainable attributes is difficult to construct, especially in the vision-based domain.…”

Section: Explaining Traditional Rl Agentsmentioning

confidence: 99%

Self-Supervised Discovering of Interpretable Features for Reinforcement Learning

Shi¹,

Huang²,

Song³

et al. 2020

Preprint

View full text Add to dashboard Cite

Deep reinforcement learning (RL) has recently led to many breakthroughs on a range of complex control tasks. However, the agent's decision-making process is generally not transparent. The lack of interpretability hinders the applicability of RL in safety-critical scenarios. In this paper, we propose a self-supervised interpretable framework, which employs a self-supervised interpretable network (SSINet) to discover and locate fine-grained causal features that constitute most evidence for the agent's decisions. We verify and evaluate our method on several Atari 2600 games as well as Duckietown. The results show that our method renders causal explanations and empirical evidences about how the agent makes decisions and why the agent performs well or badly. Moreover, our method is a flexible explanatory module that can be applied to most vision-based RL agents. Overall, our method provides valuable insight into interpretable vision-based RL.

show abstract

Conservative Q-Improvement: Reinforcement Learning for an Interpretable Decision-Tree Policy

Cited by 13 publications

References 9 publications

Evolutionary learning of interpretable decision trees

Evolutionary learning of interpretable decision trees

A Survey on Interpretable Reinforcement Learning

Self-Supervised Discovering of Interpretable Features for Reinforcement Learning

Contact Info

Product

Resources

About