Optimizing n‑gram Order of an n‑gram Based Language Identification Algorithm for 68 Written Languages

Abstract

Language identification technology is widely used in the domains of machine learning and text mining. Many researchers have achieved excellent results on a few selected European languages. However, the majority of African and Asian languages remain untested. The primary objective of this research is to evaluate the performance of our new n-gram based language identification algorithm on 68 written languages used in the European, African and Asian regions. The secondary objective is to evaluate how n-gram orders and a mix n-gram model affect the relative performance and accuracy of language identification. The n-gram based algorithm used in this paper does not depend on the n-gram frequency. Instead, the algorithm is based on a Boolean method to determine the output of matching target n-grams to training n-grams. The algorithm is designed to automatically detect the language, script and character encoding scheme of a written text. It is important to identify these three properties due to the reason that a language can be written in different types of scripts and encoded with different types of character encoding schemes. The experimental results show that in one test the algorithm achieved up to 99.59% correct identification rate on selected languages. The results also show that the performance of language identification can be improved by using a mix n-gram model of bigram and trigram. The mix n-gram model consumed less disk space and computing time, compared to a trigram model.

A word‐level language identification strategy for resource‐scarce languages

“…Also, previous research has shown that performances of similar experiments using different values of n usually peak at 3, i.e. trigrams (Choong, Mikami, Marasinghe, & Nandasara, ; Suzuki, Mikami, Ohsato, & Chubachi, ; Jhamtani, Bhogi, & Raychoudhury, ; Chittaranjan, Vyas, Bali, & Choudhury, ).…”

LanideNN: Multilingual Language Identification on Text Stream

“…All the n-gram-based ap-proaches differ primarily in the calculation of the distance between the n-gram profile of the training and test text (Selamat, 2011;Yang and Liang, 2010), or by using additional features on top of the n-gram profiles (Padma et al, 2009;Carter et al, 2013). One of the fairly robust definitions of the distance (or similarity) was proposed by Choong et al (2009) who simply check the proportion of n-gram types seen in the tested document of the most frequent n-gram types extracted from training documents for each language. The highestscoring language is then returned.…”

On Hierarchical Text Language-Identification Algorithms

“…It is a process that attempts to classify text in a language into a pre-defined set of known languages [1]. LI is the first step in text mining, information retrieval, speech processing and machine translation [2][3][4].…”