M-Arg: Multimodal Argument Mining Dataset for Political Debates with Audio and Transcripts

Abstract

Argumentation mining aims at extracting, analysing and modelling people's arguments, but large, high-quality annotated datasets are limited, and no multimodal datasets exist for this task. In this paper, we present M-Arg, a multimodal argument mining dataset with a corpus of US 2020 presidential debates, annotated through crowd-sourced annotations. This dataset allows models to be trained to extract arguments from natural dialogue such as debates using information like the intonation and rhythm of the speaker. Our dataset contains 7 hours of annotated US presidential debates, 6527 utterances and 4104 relation labels, and we report results from different baseline models, namely a text-only model, an audio-only model and multimodal models that extract features from both text and audio. With accuracy reaching 0.86 in multimodal models, we find that audio features provide added value with respect to text-only models.

Augmenting pre-trained language models with audio feature embedding for argumentation mining in political debates

“…The results were positive towards the addition of audio, although the performance was modest due to the small size of the dataset and the limitations of the text and audio representations. The only other work that considered multimodal aspects used the M-arg dataset (Mestre et al, 2021). There, the authors analyzed argumentative relations in the 2016 US presidential debates using text and audio, building an argumentation mining pipeline based on BERT embeddings for text and a combination of a Bi-LSTM and a CNN for the audio.…”

“…Here, we build upon a previous dataset (US-ElecDeb60To16) presented by Haddadan et al (2019), which contained English transcripts of the US presidential debates from 1960 to 2016 labelled with more than 29k annotations of argument components and their boundaries. We used the original videos from the debates to obtain aligned timestamps at the sentence level following the work of Mestre et al (2021), thus enabling the task of multimodal AM with a total of 28,850 aligned and annotated sentences. Concurrently to the submission of our work, Mancini et al (2022) also presented and released a multimodal dataset, using the same videos and alignment process, with 26,791 sentences.…”

“…Both datasets are complementary, although our dataset is slightly larger as we did not drop any of the debate videos (see next section). Mancini et al (2022) compare datasets and architectures from the two previously mentioned works by Lippi and Torroni (2016) and Mestre et al (2021), as well as their new dataset, finding generally positive results to the addition of audio. In our work, we report an audio feature analysis, as well as the impact of using computerized voices, and we investigate the benefit of multimodal models on both balanced and fractional small data subsets.…”

“…We aligned the transcripts with the audio using the Aeneas Forced Alignment (v. 1.7.3) tool as proposed by Mestre et al (2021). Two researchers manually checked every debate for major misalignment (and fixed them) until we obtained an almost perfectly aligned text.…”

“…We used the architectures proposed by Mestre et al (2021), which showed the potential of multimodal argumentation mining in our dataset (Figure 1). We considered two text modules based on GloVe and BERT (Devlin et al, 2018).…”