Exploring Distantly-Labeled Rationales in Neural Network Models

Abstract

Recent studies strive to incorporate various human rationales into neural networks to improve model performance, but few pay attention to the quality of the rationales. Most existing methods distribute their models' focus to distantly-labeled rationale words entirely and equally, while ignoring the potential important non-rationale words and not distinguishing the importance of different rationale words. In this paper, we propose two novel auxiliary loss functions to make better use of distantly-labeled rationales, which encourage models to maintain their focus on important words beyond labeled rationales (PINs) and alleviate redundant training on non-helpful rationales (NoIRs). Experiments on two representative classification tasks show that our proposed methods can push a classification model to effectively learn crucial clues from nonperfect rationales while maintaining the ability to spread its focus to other unlabeled important words, thus significantly outperform existing methods.

Using Interpretation Methods for Model Enhancement

“…We also discuss how the optimization of the task-specific loss and the new loss should be coordinated. For previous methods utilizing gradient-based interpretation for model enhancement (Huang et al, 2021;Ghaeini et al, 2019), we show that they can be seen as instances of our framework.…”

“…In Ghaeini et al (2019)'s work, f is a function that takes the sum of the gradients of each input embedding dimension and D is to take the sum of the gradients of rationale words. In Huang et al (2021)'s work, f is the L 1 norm that sums up the absolute value of gradients over the input embedding dimensions and D is designed in various ways to give rationale words higher attribution scores.…”

“…This intuitive idea, however, has not been fully explored. There are only a few previous studies based on the gradientbased category of interpretation methods (Huang et al, 2021;Ghaeini et al, 2019;Liu and Avci, 2019) and they neither compare the utilization of different interpretation methods on model enhancement nor experiment on a comprehensive range of tasks.…”