MiNgMatch—A Fast N-gram Model for Word Segmentation of the Ainu Language

Abstract

Word segmentation is an essential task in automatic language processing for languages where there are no explicit word boundary markers, or where space-delimited orthographic words are too coarse-grained. In this paper we introduce the MiNgMatch Segmenter-a fast word segmentation algorithm, which reduces the problem of identifying word boundaries to finding the shortest sequence of lexical n-grams matching the input text. In order to validate our method in a low-resource scenario involving extremely sparse data, we tested it with a small corpus of text in the critically endangered language of the Ainu people living in northern parts of Japan. Furthermore, we performed a series of experiments comparing our algorithm with systems utilizing state-of-the-art lexical n-gram-based language modelling techniques (namely, Stupid Backoff model and a model with modified Kneser-Ney smoothing), as well as a neural model performing word segmentation as character sequence labelling. The experimental results we obtained demonstrate the high performance of our algorithm, comparable with the other best-performing models. Given its low computational cost and competitive results, we believe that the proposed approach could be extended to other languages, and possibly also to other Natural Language Processing tasks, such as speech recognition.

Ainu–Japanese Bi-directional Neural Machine Translation: A Step Towards Linguistic Preservation of Ainu, An Under-Resourced Indigenous Language in Japan

“…The exploration of advanced computational methods like NLP and MT in the context of language revival is increasingly gaining traction. The pioneering work of Nowakowski et al [2019] introduced an efficient n-gram model for the segmentation of Ainu words, demonstrating the capabilities of computational techniques to enhance the study and accessibility of the Ainu language. Following this, Nowakowski [2020] took further strides by developing a digital corpus alongside fundamental language technologies for Ainu.…”

“…Foundational work by researchers like Nowakowski et al [2019], who developed the Mingmatch-a n-gram model for segmenting Ainu words, and Matsuura et al [2020b], who compiled an Ainu folklore speech corpus, provides critical underpinnings for this study. Building upon these seminal contributions, this project seeks to construct a comprehensive NLP model that utilizes the Marian MT as its core for translating between Ainu and Japanese.…”