Building and curating conversational corpora for diversity-aware language science and technology

Abstract

We present a pipeline and tools to build a maximally natural data set of conversational interaction that covers 66 languages and varieties from 32 phyla. We describe the curation and compilation process moving from diverse language documentation corpora to a unified format and describe an open-source tool 'convo-parse' to help in quality control and assessment of conversational data. We conclude with two case studies of how diverse data sets can inform interactional linguistics and speech recognition technology and thus contribute to broadening the empirical foundations of language sciences and technologies of the future.

From text to talk: Harnessing conversational corpora for humane and diversity-aware language technology

“…Our focus is specifically on corpora of informal conversations among co-present participants, transcribed and time-aligned at the level of conversational turns. Details of our curation and analysis pipeline are described in the Appendix and in Liesenfeld & Dingemanse (2022). While it is impossible to exhaustively list or estimate the size of extant conversational corpora, the qualitycontrolled subset we consider here represents 63 languages from 26 language families (Figure 1), and amounts to over 800 hours of talk produced by over 11.000 partipants, segmented into over 1.6 million turns (9.3 million words) (Figure 2).…”

“…With regard to data usage, the corpora considered here are only those for which contributors have granted access openly or to all registered users, usually for research purposes. We cannot redistribute the dataset directly, but have strived to document the process of curation in sufficient detail to enable others to register and access the data (Liesenfeld and Dingemanse, 2022). With regard to technological applications, as with any technology, there is potential for helpful as well as harmful uses (Hovy and Spruit, 2016) and we side with Levow et al (2021) in stressing the need for computational linguists to work closely with language communities in maximising helpful uses and minimising harmful ones.…”