MARTA: Leveraging Human Rationales for Explainable Text Classification

Abstract

Explainability is a key requirement for text classification in many application domains ranging from sentiment analysis to medical diagnosis or legal reviews. Existing methods often rely on "attention" mechanisms for explaining classification results by estimating the relative importance of input units. However, recent studies have shown that such mechanisms tend to mis-identify irrelevant input units in their explanation. In this work, we propose a hybrid human-AI approach that incorporates human rationales into attention-based text classification models to improve the explainability of classification results. Specifically, we ask workers to provide rationales for their annotation by selecting relevant pieces of text. We introduce MARTA, a Bayesian framework that jointly learns an attention-based model and the reliability of workers while injecting human rationales into model training. We derive a principled optimization algorithm based on variational inference with efficient updating rules for learning MARTA parameters. Extensive validation on real-world datasets shows that our framework significantly improves the state of the art both in terms of classification explainability and accuracy.

Explainable Text Classification Techniques in Legal Document Review: Locating Rationales without Using Human Annotated Training Text Snippets

“…An explanation of a responsive document is one or more text snippets, referred to as rationale in [5], in the responsive document. Explainable predictive coding sets out to build a method to estimate the following probability:…”

“…Recent research found that a prediction-based approach is often used to identify snippets of text as an explanation for the classification of a document. A text snippet that explains the classification of a document is called a 'rationale' for the document in [4] [5]. From a machine learning perspective, annotated training text snippets provide more effective labeled input due to their targeted evidence relating to the relevance of the decision.…”

Improving Retrieval in Theme-specific Applications using a Corpus Topical Taxonomy

“…To construct the topical structure reflecting theme-specific user interests, taxonomies are built upon a foundational seed taxonomy rooted in human knowledge of the application [1,14,26]. An example is the fields of study in the academic domain [46], which embodies researchers' inclination to organize academic concepts and studies.…”

“…A corpus topical taxonomy outlines the latent topic hierarchy within the corpus as a tree structure, where each node is a topic class represented by a cluster of semantically coherent terms describing the topic, as shown in Figure 1. Recent taxonomy construction studies [1,14,26] have effectively reflected user-interested aspects, drawing from a foundational seed taxonomy rooted in human knowledge of the application (e.g., fields of study from Mircosoft Academic [46]). The constructed taxonomy can be subsequently employed to provide additional clues to link queries and documents by discerning their topical relatedness and supplementing the missing contexts.…”

Online Target Localization Using Adaptive Belief Propagation in the HMM Framework

“…where α h t ∈ R >0 and β h t ∈ R >0 are the parameters of the distribution at time t. The beta distribution, defined within the interval [0 1] ⊆ R, is well-suited for statistical modeling of human detection reliability and has been previously used in other contexts such as human-aided text classification [32]. For different values of α h t and β h t , we can capture different levels of reliability as shown in Fig 4 on the right (low, mediocre, high).…”