The use of annotations to explain labels: Comparing results from a human‐rater approach to a deep learning approach

Abstract

Background Deep learning methods, where models do not use explicit features and instead rely on implicit features estimated during model training, suffer from an explainability problem. In text classification, saliency maps that reflect the importance of words in prediction are one approach toward explainability. However, little is known about whether the salient words agree with those identified by humans as important. Objectives The current study examines in‐line annotations from human annotators and saliency map annotations from a deep learning model (ELECTRA transformer) to understand how well both humans and machines provide evidence for their assigned label. Methods Data were responses to test items across a mix of United States subjects, states, and grades. Humans were trained to annotate responses to justify a crisis alert label and two model interpretability methods (LIME, Integrated Gradients) were used to obtain engine annotations. Human inter‐annotator agreement and engine agreement with the human annotators were computed and compared. Results and Conclusions Human annotators agreed with one another at similar rates to those observed in the literature on similar tasks. The annotations derived using the integrated gradients (IG) agreed with human annotators at higher rates than LIME on most metrics; however, both methods underperformed relative to the human annotators. Implications Saliency map‐based engine annotations show promise as a form of explanation, but do not reach human annotation agreement levels. Future work should examine the appropriate unit for annotation (e.g., word, sentence), other gradient based methods, and approaches for mapping the continuous saliency values to Boolean annotations.

Artificial intelligence on the advance to enhance educational assessment: Scientific clickbait or genuine gamechanger?

“…Although in a very different application context, Lottridge et al (2023) followed a similar purpose and methodological setup, which led to similar results to those of Gombert et al (2023) regarding the lack of correspondence between what machines and humans identified at the output level as the determining factors in the classification of responses. The purpose of the built classifier is to identify students at risk—due to anxiety, abuse, suicidal tendencies, and the like—who mention their critical situation in their text responses when participating in a large‐scale assessment.…”

“…Similar to the study by Lottridge et al (2023), the study by Andersen et al (2023) was situated in large-scale assessments where large amounts of human resources are devoted to manually coding constructed responses. To keep the human in the loop and in control, Andersen et al (2023) developed a method that dynamically varies the amount of automatically scored short text responses, depending on the performance of the automatic system.…”

“…Somewhat surprisingly, the scope of all papers that addressed text data involves some kind of focus on the human. While Gombert et al (2023) and Lottridge et al (2023) aimed at achieving post hoc explainability of classifiers for humans' interpretations, the other three papers directly involved human actors at some point in the envisaged process. Let us start with the explainability‐focused papers.…”

The Explainability of Transformers: Current Status and Directions

“…El Zini et al employed LIME, Anchors, and SHAP when introducing a new dataset to evaluate the performances of different models for sentiment analysis [136]. Lottridge et al compared annotations from humans with respect to explanations provided by both LIME and Integrated Gradients within the scope of crisis alert identification [137]. Arashpour et al compared a wide range of explainability methods falling into the classes of perturbation-based methods and gradient-based methods (Integrated Gradients, Gradient SHAP, Occlusion, the Fast Gradient Sign Method, Projected Gradient Descent, Minimal Perturbation, and Feature Ablation) for waste categorization in images [138].…”

Transforming Assessment: The Impacts and Implications of Large Language Models and Generative AI

“…Several automated scoring engines have been used in operations for scoring short‐constructed responses (e.g., Braun et al., 2006; Heilman & Madnani, 2015), long essays (e.g., Attali & Burstein, 2006), and mathematical equations (Fife, 2017). Recently, the transformer architecture underlying most LLMs has shown excellent performance in automated scoring of short‐constructed responses, essays, and speech (Lottridge et al., 2022a; Lottridge et al., 2022b; Lottridge et al., 2022c). In fact, the winners of the 2021 NAEP automated scoring competition all used the transformer architecture (https://github.com/NAEP‐AS‐Challenge/reading‐prediction/blob/main/results.md).…”