The IUCL+ System: Word-Level Language Identification via Extended Markov Models

King, Levi; Baucom, Eric; Gilmanov, Timur; Kübler, Sandra; Whyatt, Dan; Maier, Wolfgang; Rodrigues, Paul

doi:10.3115/v1/w14-3912

Cited by 9 publications

(8 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As an example, some gapped 4-grams from the word Sterneberg would be Senb, tree, enbr , and reeg. King et al (2014b) used character n-grams as a backoff from Markovian word n-grams. Shrestha (2014) used the frequencies of word initial n-grams ranging from 3 to the length of the word minus 1.…”

Section: Character Repetitionmentioning

confidence: 99%

Automatic Language Identification in Texts: A Survey

Jauhiainen

Lui²,

Zampieri³

et al. 2019

jair

104

View full text Add to dashboard Cite

Language identification ("LI") is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used in the LI literature. We describe the features and methods using a unified notation, to make the relationships between methods clearer. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.LI as a task predates computational methods -the earliest interest in the area was motivated by the needs of translators, and simple manual methods were developed to quickly identify documents in specific languages. The earliest known work to describe a functional LI program for text is by Mustonen (1965), a statistician, who used multiple discriminant analysis to teach a computer how to distinguish, at the word level, between English, Swedish and Finnish. Mustonen compiled a list of linguistically-motivated character-based features, and trained his language identifier on 300 words for each of the three target languages. The training procedure created two discriminant functions, which were tested with 100 words for each language. The experiment resulted in 76% of the words being correctly classified; even by current standards this percentage would be seen as acceptable given the small amount of training material, although the composition of training and test data is not clear, making the experiment unreproducible.In the early 1970s, Nakamura (1971) considered the problem of automatic LI. According to Rau (1974) and the available abstract of Nakamura's article, 1 his language identifier was able to distinguish between 25 languages written with the Latin alphabet. As features, the method used the occurrence rates of characters and words in each language. From the abstract it seems that, in addition to the frequencies, he used some binary presence/absence features of particular characters or words, based on manual LI. Rau (1974) wrote his master's thesis "Language Identification by Statistical Analysis" for the Naval Postgraduate School at Monterey, California. The continued interest and the need to use LI of text in military intelligence settings is evidenced by the recent articles of, for example, Rafidha Rehiman et al. (2013), Rowe et al. (2013), and Voss et al. (2014. As features for LI, Rau (1974) used, e.g., the relative frequencies of characters and character bigrams. With a majority vote classifier ensemble of seven classifiers using Kolmogor-Smirnov's Test of Goodness of Fit and Yule's characteristic (K), he managed...

show abstract

Section: Character Repetitionmentioning

confidence: 99%

Automatic Language Identification in Texts: A Survey

Jauhiainen

Lui²,

Zampieri³

et al. 2019

jair

104

View full text Add to dashboard Cite

show abstract

“…• IUCL: The best results obtained by King et al (2014); • IIIT: The best results obtained by Jain and Bhat (2014); • CMU: The best results obtained by Lin et al (2014); • MSR-India: The best results obtained by Chittaranjan et al ( 2014); • AIDA: The best results obtained by us using the older version AIDA .…”

Section: Token Level Baselinesmentioning

confidence: 92%

“…Their system outperforms Zaidan and Callison-Burch (2011) and Elfardy and Diab (2013), achieving a classification accuracy of 89%, 79%, and 88% on the same Egyptian, Levantine and Gulf datasets. For token level dialect identification, King et al (2014) use a language-independent approach that utilizes character n-gram probabilities, lexical probabilities, word label transition probabilities and existing named entity recognition tools within a Markov model framework.…”

Section: Related Workmentioning

confidence: 99%

AIDA2: A Hybrid Approach for Token and Sentence Level Dialect Identification in Arabic

Al-Badrashiny¹,

Elfardy²,

Diab³

2015

Proceedings of the Nineteenth Conference on Computational Natural Language Learning

View full text Add to dashboard Cite

In this paper, we present a hybrid approach for performing token and sentence levels Dialect Identification in Arabic. Specifically we try to identify whether each token in a given sentence belongs to Modern Standard Arabic (MSA), Egyptian Dialectal Arabic (EDA) or some other class and whether the whole sentence is mostly EDA or MSA. The token level component relies on a Conditional Random Field (CRF) classifier that uses decisions from several underlying components such as language models, a named entity recognizer and and a morphological analyzer to label each word in the sentence. The sentence level component uses a classifier ensemble system that relies on two independent underlying classifiers that model different aspects of the language. Using a featureselection heuristic, we select the best set of features for each of these two classifiers. We then train another classifier that uses the class labels and the confidence scores generated by each of the two underlying classifiers to decide upon the final class for each sentence. The token level component yields a new state of the art F-score of 90.6% (compared to previous state of the art of 86.8%) and the sentence level component yields an accuracy of 90.8% (compared to 86.6% obtained by the best state of the art system).

show abstract

“…But here as well, the small number of labeled instances makes it hard to draw strong conclusions. (Barman et al, 2014) 0.84 0.85 0.31 0.82 0.03 0 0.823 (Chittaranjan et al, 2014) 0.94 0.86 0.14 0.83 0 0 0.824 (King et al, 2014) 0.84 0.85 0.35 0.81 0 0 0.828 (Bar and Dershowitz, 2014) 0.85 0.87 0.37 0.83 0.03 0 0.839 Table 4: Performance results on language identification at the token level. A '-' indicates there were no tokens of this class in the test set.…”

Section: Resultsmentioning

confidence: 99%

“…However, according to the system descriptions provided, not all systems used encoding information. The best performing systems for MAN-EN are (King et al, 2014) and (Chittaranjan et al, 2014). The former slightly outperformed the latter at the Tweet level (see Figure 1a) task while the opposite was true at the token level (see Table 4 rows 4 and 5).…”

Section: Resultsmentioning

confidence: 99%

Overview for the First Shared Task on Language Identification in Code-Switched Data

Solorio¹,

Blair²,

Maharjan³

et al. 2014

Proceedings of the First Workshop on Computational Approaches to Code Switching

152

191

View full text Add to dashboard Cite

We present an overview of the first shared task on language identification on codeswitched data.The shared task included code-switched data from four language pairs: Modern Standard ArabicDialectal Arabic (MSA-DA), MandarinEnglish (MAN-EN), Nepali-English (NEP-EN), and Spanish-English (SPA-EN). A total of seven teams participated in the task and submitted 42 system runs. The evaluation showed that language identification at the token level is more difficult when the languages present are closely related, as in the case of MSA-DA, where the prediction performance was the lowest among all language pairs. In contrast, the language pairs with the higest F-measure where SPA-EN and NEP-EN. The task made evident that language identification in code-switched data is still far from solved and warrants further research.

show abstract

The IUCL+ System: Word-Level Language Identification via Extended Markov Models

Cited by 9 publications

References 6 publications

Automatic Language Identification in Texts: A Survey

Automatic Language Identification in Texts: A Survey

AIDA2: A Hybrid Approach for Token and Sentence Level Dialect Identification in Arabic

Overview for the First Shared Task on Language Identification in Code-Switched Data

Contact Info

Product

Resources

About