Seeing More Than Whitespace — Tokenisation and Disambiguation in a North Sámi Grammar Checker

Abstract

Communities of lesser resourced languages like North Sámi benefit from language tools such as spell checkers and grammar checkers to improve literacy. Accurate error feedback is dependent on well-tokenised input, but traditional tokenisation as shallow preprocessing is inadequate to solve the challenges of real-world language usage. We present an alternative where tokenisation remains ambiguous until we have linguistic context information available. This lets us accurately detect sentence boundaries, multiwords and compound error detection. We describe a North Sámi grammar checker with such a tokenisation system, and show the results of its evaluation.

GiellaLT: an infrastructure for rule-based language technology tool development

“…In our tokenisation, sentence boundary detection is treated as a special case of ambiguous tokenisation, and solved in the same way, approaching nearperfect sentence boundary identification, cf. Wiechetek et al (2019b).…”

“…There is also support for neural network based models of spellchecking (Kaalep et al, 2022), this is however in its current stage still not up to par with the traditional weighted finitestate models given the current error corpus sizes. Since 2019 the GiellaLT infrastructure sup ports building grammar checkers (Wiechetek et al, 2019a) and these are available for some of the Sámi languages already. Another highlevel tool avail able within the GiellaLT infrastructure is machine translation.…”

ItaGLAM: A corpus of Cultural Communication on Twitter during the Pandemic

“…words that are written apart and should be a compound) in GramDivvun in two ways. Firstly, we compare last year's results in Wiechetek (2019a) with a newer version of Gram-Divvun, from now on referred to as the Nodalida-corpus. Last year's results are based on version r183544 (Wiechetek et al, 2019a) .…”

“…Firstly, we compare last year's results in Wiechetek (2019a) with a newer version of Gram-Divvun, from now on referred to as the Nodalida-corpus. Last year's results are based on version r183544 (Wiechetek et al, 2019a) . The new results are based on version r28510 of Gram-Divvun.…”

Rules Ruling Neural Networks – Neural vs. Rule-Based Grammar Checking for a Low Resource Language

“…Rules have been used and are in a wide-spread use in the context of endangered Uralic languages. There is recent work on grammar checking for North Sámi (Wiechetek et al, 2019a) and spell checking for Skolt Sámi (Trosterud and Moshagen, 2021). Other rule-based approaches to grammar checking are extensively described in Wiechetek (2017).…”