Reaching beneath the tip of the iceberg: A guide to the Freiburg Multimodal Interaction Corpus

Abstract

Most corpora tacitly subscribe to a speech-only view filtering out anything that is not a ‘word’ and transcribing the spoken language merely orthographically despite the fact that the “speech-only view on language is fundamentally incomplete” (Kok 2017, 2) due to the deep intertwining of the verbal, vocal, and kinesic modalities (Levinson and Holler 2014). This article introduces the Freiburg Multimodal Interaction Corpus (FreMIC), a multimodal and interactional corpus of unscripted conversation in English currently under construction. At the time of writing, FreMIC comprises (i) c. 29 h of video-recordings transcribed and annotated in detail and (ii) automatically (and manually) generated multimodal data. All conversations are transcribed in ELAN both orthographically and using Jeffersonian conventions to render verbal content and interactionally relevant details of sequencing (e.g. overlap, latching), temporal aspects (pauses, acceleration/deceleration), phonological aspects (e.g. intensity, pitch, stretching, truncation, voice quality), and laughter. Moreover, the orthographic transcriptions are exhaustively PoS-tagged using the CLAWS web tagger (Garside and Smith 1997). ELAN-based transcriptions also provide exhaustive annotations of re-enactments (also referred to as (free) direct speech, constructed dialogue, etc.) as well as silent gestures (meaningful gestures that occur without accompanying speech). The multimodal data are derived from psychophysiological measurements and eye tracking. The psychophysiological measurements include, inter alia, electrodermal activity or GSR, which is indicative of emotional arousal (e.g. Peräkylä et al. 2015). Eye tracking produces data of two kinds: gaze direction and pupil size. In FreMIC, gazes are automatically recorded using the area-of-interest technology. Gaze direction is interactionally key, for example, in turn-taking (e.g. Auer 2021) and re-enactments (e.g. Pfeiffer and Weiss 2022), while changes in pupil size provide a window onto cognitive intensity (e.g. Barthel and Sauppe 2019). To demonstrate what opportunities FreMIC’s (combination of) transcriptions, annotations, and multimodal data open up for research in Interactional (Corpus) Linguistics, this article reports on interim results derived from work-in-progress.

Word frequency and cognitive effort in turns-at-talk: turn structure affects processing load in natural conversation

“…Hoffmann et al, 2008 ) and (ii) the Freiburg Multimodal Interaction Corpus (FreMIC; cf. Rühlemann and Ptak, 2023 ).…”

“…This study aims to fill this gap by investigating the structure of frequency in turns drawing on data from two corpora: the British National Corpus (BNC) and the Freiburg Multimodal Interaction Corpus (FreMIC; see Section 2.1 Corpora ). The FreMIC, a novel corpus (Rühlemann and Ptak, 2023 ), holds information about interlocutors' pupil sizes that were recorded alongside the conversations. While pupil size is affected by a number of factors, including lighting conditions, drug consumption, pathological states, and emotional arousal, it is known to also reflect processing intensity (Beatty, 1982 ; Beatty and Lucero-Wagoner, 2000 ; Sirois and Brisson, 2014 ; cf.…”

“… 3 The description of FreMIC that follows will focus exclusively on the aspects and components that are of relevance for the present analyses; for a comprehensive overview see Rühlemann and Ptak ( 2023 ). The construction of FreMIC has been facilitated by a grant by the Deutsche Forschungsgemeinschaft (DFG); grant number 497779797.…”