Harnessing Indigenous Tweets: The Reo Māori Twitter corpus

Abstract

Te reo Māori, the Indigenous language of Aotearoa New Zealand, is a distinctive feature of the nation’s cultural heritage. This paper documents our efforts to build a corpus of 79,000 Māori-language tweets using computational methods. The Reo Māori Twitter (RMT) Corpus was created by targeting Māori-language users identified by the Indigenous Tweets website, pre-processing their data and filtering out non-Māori tweets, together with other sources of noise. Our motivation for creating such a resource is three-fold: (1) it serves as a rich and unique dataset for linguistic analysis of te reo Māori on social media; (2) it can be used as training data to develop and augment Natural Language Processing (NLP) tools with robust, real-world Māori-language applications; and (3) it will potentially promote awareness of, and encourage positive interaction with, the growing community of Māori tweeters, thereby increasing the use and visibility of te reo Māori in an online environment. While the corpus captures data from 2007 to 2020, our analysis shows that the number of tweets in the RMT Corpus peaked in 2014, and the number of active tweeters peaked in 2017, although at least 600 users were still active in 2020. To the best of our knowledge, the RMT Corpus is the largest publicly-available collection of social media data containing (almost) exclusively Māori text, making it a useful resource for language experts, NLP developers and Indigenous researchers alike.

The development of a labelled te reo Māori–English bilingual database for language technology

“…The language detection model was also extended using the labelled Hansard corpus to detect Māori-English code-switching points in the text. This was also tested on the labelled Hansard database, and the Reo Māori Tweets corpus [16]. The labelled Hansard corpus is currently being used to improve language detection and code-switch detection models.…”

“…Since code-switching appears more in informal contexts such as social media, future research should focus on developing labelled corpora containing social media text. Some attempts in this regard have been reported [16,17], but more rigorous testing of these corpora in language processing applications is still to be done.…”

“…The Reo Māori Twitter (RMT) Corpus [15,16], a corpus of Māori-language tweets, and the Māori Loanwords on Twitter (MLT) corpus [17] are Māori databases that contain informal tweets data. These corpora are used extensively for linguistic analysis.…”

Strategic illiteracies: the long game of technology refusal and disconnection

“…And in sheer self-protection, foreign traders faced with Roman customs-men must have learned to keep records"(2004, p. 1157). 6 Te Reo M aori, for example, is thriving both in broadcast media(Middleton, 2020) and online(Trye et al, 2022).7 Japan has its own complex history of orthographic resistance (e.g.,Heinrich, 2012;Takayama, 1995;Unger, 1996). 8 New writing systems do not necessarily lead to extinction of old, but script shifts and disappearances have often happened via biscriptalism(Houston, 2004, pp.…”