Applying Grapheme, Word, and Syllable Information for Language Identification in Code Switching Sentences

Yeong, Yin-Lai; Tan, Tien-Ping

doi:10.1109/ialp.2011.34

Cited by 8 publications

(9 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Yeong and Tan (2010) used Markovian syllable bigrams for LI between Malay and English. Later Yeong and Tan (2011) also experimented with syllable uni-and trigrams. Murthy and Kumar (2006) used the most frequent as well as the most discriminating Indian script syllables, called aksharas.…”

Section: Morphemes Syllables and Chunksmentioning

confidence: 99%

“…in the case of the Hangul or Thai writing systems). Later, Yeong and Tan (2011) also used grapheme uniand trigrams. Yeong and Tan (2011) achieved their best results combining word unigrams and syllable bigrams with a grapheme back-off.…”

Section: Morphemes Syllables and Chunksmentioning

confidence: 99%

“…Later, Yeong and Tan (2011) also used grapheme uniand trigrams. Yeong and Tan (2011) achieved their best results combining word unigrams and syllable bigrams with a grapheme back-off. used the MADAMIRA toolkit for D3 decliticization and then used D3-token 5-grams.…”

Section: Morphemes Syllables and Chunksmentioning

confidence: 99%

See 2 more Smart Citations

Automatic Language Identification in Texts: A Survey

Jauhiainen

Lui²,

Zampieri³

et al. 2019

jair

104

View full text Add to dashboard Cite

Language identification ("LI") is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used in the LI literature. We describe the features and methods using a unified notation, to make the relationships between methods clearer. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.LI as a task predates computational methods -the earliest interest in the area was motivated by the needs of translators, and simple manual methods were developed to quickly identify documents in specific languages. The earliest known work to describe a functional LI program for text is by Mustonen (1965), a statistician, who used multiple discriminant analysis to teach a computer how to distinguish, at the word level, between English, Swedish and Finnish. Mustonen compiled a list of linguistically-motivated character-based features, and trained his language identifier on 300 words for each of the three target languages. The training procedure created two discriminant functions, which were tested with 100 words for each language. The experiment resulted in 76% of the words being correctly classified; even by current standards this percentage would be seen as acceptable given the small amount of training material, although the composition of training and test data is not clear, making the experiment unreproducible.In the early 1970s, Nakamura (1971) considered the problem of automatic LI. According to Rau (1974) and the available abstract of Nakamura's article, 1 his language identifier was able to distinguish between 25 languages written with the Latin alphabet. As features, the method used the occurrence rates of characters and words in each language. From the abstract it seems that, in addition to the frequencies, he used some binary presence/absence features of particular characters or words, based on manual LI. Rau (1974) wrote his master's thesis "Language Identification by Statistical Analysis" for the Naval Postgraduate School at Monterey, California. The continued interest and the need to use LI of text in military intelligence settings is evidenced by the recent articles of, for example, Rafidha Rehiman et al. (2013), Rowe et al. (2013), and Voss et al. (2014. As features for LI, Rau (1974) used, e.g., the relative frequencies of characters and character bigrams. With a majority vote classifier ensemble of seven classifiers using Kolmogor-Smirnov's Test of Goodness of Fit and Yule's characteristic (K), he managed...

show abstract

Section: Morphemes Syllables and Chunksmentioning

confidence: 99%

Section: Morphemes Syllables and Chunksmentioning

confidence: 99%

See 1 more Smart Citation

Automatic Language Identification in Texts: A Survey

Jauhiainen

Lui²,

Zampieri³

et al. 2019

jair

104

View full text Add to dashboard Cite

show abstract

“…Thus, we renounce morphological processing as described in Yeong and Tan (2011) and prosodic features since we are working with written text.…”

Section: Related Workmentioning

confidence: 99%

“…Nepali-English, Mandarin-English or Spanish-English. Yeong and Tan (2011) use morphological structure and sequence of syllables in Malay-English sentences to identify language. Barman et al (2014) investigate mixed text including three languages: Bengali, English and Hindi.…”

Section: Related Workmentioning

confidence: 99%

Code-Switching Ubique Est - Language Identification and Part-of-Speech Tagging for Historical Mixed Text

Schulz¹,

Keller²

2016

Proceedings of the 10th SIGHUM Workshop on Language Technology For Cultural Heritage, Social Sciences, and Humanities

View full text Add to dashboard Cite

In this paper, we describe the development of a language identification system and a part-of-speech tagger for Latin-Middle English mixed text. To this end, we annotate data with language IDs and Universal POS tags (Petrov et al., 2012). As a classifier, we train a conditional random field classifier for both sub-tasks, including features generated by the TreeTagger models of both languages. The focus lies on both a general and a task-specific evaluation. Moreover, we describe our effort concerning beyond proof-of-concept implementation of tools and towards a more task-oriented approach, showing how to apply our techniques in the context of Humanities research.

show abstract

Usefulness of Graphemes in Word-Level Language Identification in Code-Mixed Text

Jain

Agarwal

2022

Lecture Notes in Networks and Systems

View full text Add to dashboard Cite

Applying Grapheme, Word, and Syllable Information for Language Identification in Code Switching Sentences

Cited by 8 publications

References 1 publication

Automatic Language Identification in Texts: A Survey

Automatic Language Identification in Texts: A Survey

Code-Switching Ubique Est - Language Identification and Part-of-Speech Tagging for Historical Mixed Text

Usefulness of Graphemes in Word-Level Language Identification in Code-Mixed Text

Contact Info

Product

Resources

About