Automatically Generated Explanations for Markov Decision Processes

Abstract

Explaining policies of Markov Decision Processes (MDPs) is complicated due to their probabilistic and sequential nature. We present a technique to explain policies for factored MDPs by populating a set of domain-independent templates. We also present a mechanism to determine a minimal set of templates that, viewed together, completely justify the policy. These explanations can be generated automatically at run-time with no additional effort required from the MDP designer. We demonstrate this technique using the problems of advising undergraduate students in their course selection and assisting people with dementia in completing the task of handwashing. We also evaluate these automatically generated explanations for course-advising through a user study involving students.

An English-Language Argumentation Interface for Explanation Generation with Markov Decision Processes in the Domain of Academic Advising

“…An example of an explanation of this form, in which only one rewarding scenario need be considered is, taken verbatim from [Khan et al 2009]: Action TakeCS343 & CS448 is the best action because: It is likely to take you to CoursesCompleted = 6, TermNumber = Final about 0.86 times, which is as high as any other action.…”

“…While this particular explanation consists of only a single template, in general the full explanation for π * (s) consists of the minimum number of templates required to explain some portion of the utility, V MSE , such that Q(s, π * (s)) ≥ V MSE > Q(s, a), ∀a = π * (s) [Khan et al 2011]. We refer to an approach such as this, which focuses on explaining some suitable portion of the utility function, as a "most coverage" approach.…”

“…However, these algorithms have the advantage of being entirely domain-independent. A good example of such a system is Khan et al's minimal sufficient explanations [Khan et al 2011], which decomposes the utility of the recommended action in the current state in terms of the expected occupancy frequencies of desired future scenarios (partial assignments of state variables). Explanations are then presented as a set of sentence templates populated by action and variable names.…”

“…[Khan et al 2009] The occupancy frequency of state s , starting in state s 0 and executing policy π is…”

“…Unfortunately, this set of possible futures is a large object with many potential branches that is difficult to understand, even for sophisticated users. The complex nature of possible futures and their probabilities prevents many end users from trusting, understanding, and implementing the plans generated from MDP policies [Khan et al 2011].…”