Active Learning and Visual Analytics for Stance Classification with ALVA

Abstract

The automatic detection and classification of stance (e.g., certainty or agreement) in text data using natural language processing and machine-learning methods creates an opportunity to gain insight into the speakers’ attitudes toward their own and other people’s utterances. However, identifying stance in text presents many challenges related to training data collection and classifier training. To facilitate the entire process of training a stance classifier, we propose a visual analytics approach, called ALVA, for text data annotation and visualization. ALVA’s interplay with the stance classifier follows an active learning strategy to select suitable candidate utterances for manual annotaion. Our approach supports annotation process management and provides the annotators with a clean user interface for labeling utterances with multiple stance categories. ALVA also contains a visualization method to help analysts of the annotation and training process gain a better understanding of the categories used by the annotators. The visualization uses a novel visual representation, called CatCombos, which groups individual annotation items by the combination of stance categories. Additionally, our system makes a visualization of a vector space model available that is itself based on utterances. ALVA is already being used by our domain experts in linguistics and computational linguistics to improve the understanding of stance phenomena and to build a stance classifier for applications such as social media monitoring.

Visually-Enabled Active Deep Learning for (Geo) Text and Image Classification: A Review

“…As a result, there remain several impediments to broader adoption of M&DL, along with a range of concerns about potential negative outcomes related to the explainability of results produced. We agree here with a range of authors who have pointed to the need for human-in-the-loop strategies to both improve performance of the methods for complex problems and to increase explainability of the methods and their results [2,4,5,11,[13][14][15]. There is a clear need for methods that allow human decision-makers to assess when to accept those results and when to treat them with caution or even skepticism.…”

“…The primary focus of research in M&DL has thus far been accurate results, often at the expense of human understanding of how the results were achieved [2][3][4][5][6]. However, accurate results often depend on building large human-generated training data sets that can be expensive in both financial and person cost to create [7][8][9][10][11][12][13]. As a result, there remain several impediments to broader adoption of M&DL, along with a range of concerns about potential negative outcomes related to the explainability of results produced.…”

Evaluating stance-annotated sentences from the Brexit Blog Corpus: A quantitative linguistic analysis

“…Two annotators, one who is a professional translator with a Licentiate degree in English Linguistics and the other one with a PhD in Computational Linguistics, carried out the annotations independently of one another. For the ALVA annotation tool, see Kucher et al (2016aKucher et al ( , 2017.…”

Detection of Stance and Sentiment Modifiers in Political Blogs

“…It was also ensured that no duplicates from the gold standard were included in the pool of unlabelled data. The annotation of the actively selected sentences was performed with an annotation tool [9] specifically designed for this task. Sentences selected for annotation were presented to the annotator, who classified them according to the seven modifying categories included in the study.…”