Identification of Code-Switched Sentences and Words Using Language Modeling Approaches

Yu, Liang-Chih; He, Wei; Chien, Wei Nan; Tseng, Yuen-Hsien

doi:10.1155/2013/898714

Cited by 7 publications

(8 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A good deal of work has been done on detecting code-switched segments in speech data (Chan et al, 2004;Lyu et al, 2006). It is seen that language modelling techniques have shown promise earlier, such as in Yu et al (2013), the experiment on Mandarin-Taiwanese sentences show a high accuracy in terms of detecting code-switched sentences.…”

Section: Related Workmentioning

confidence: 98%

Subsegmental language detection in Celtic language text

Minocha¹,

Tyers²

2014

Proceedings of the First Celtic Language Technology Workshop

View full text Add to dashboard Cite

show abstract

Section: Related Workmentioning

confidence: 98%

Subsegmental language detection in Celtic language text

Minocha¹,

Tyers²

2014

Proceedings of the First Celtic Language Technology Workshop

View full text Add to dashboard Cite

show abstract

“…The term “code-switching corpus” refers to a body of text consisting of two or more languages under study (Yu et al 2013). Because Arabic and English are the primary languages in this research, we built an Arabic–English code-switching corpus.…”

Section: New Arabic Corporamentioning

confidence: 99%

Compression versus traditional machine learning classifiers to detect code-switching in varieties and dialects: Arabic as a case study

et al. 2020

View full text Add to dashboard Cite

The occurrence of code-switching in online communication, when a writer switches among multiple languages, presents a challenge for natural language processing tools, since they are designed for texts written in a single language. To answer the challenge, this paper presents detailed research on ways to detect code-switching in Arabic text automatically. We compare the prediction by partial matching (PPM) compression-based classifier, implemented in Tawa, and a traditional machine learning classifier sequential minimal optimization (SMO), implemented in Waikato Environment for Knowledge Analysis, working specifically on Arabic text taken from Facebook. Three experiments were conducted in order to: (1) detect code-switching among the Egyptian dialect and English; (2) detect code-switching among the Egyptian dialect, the Saudi dialect, and English; and (3) detect code-switching among the Egyptian dialect, the Saudi dialect, Modern Standard Arabic (MSA), and English. Our experiments showed that PPM achieved a higher accuracy rate than SMO with 99.8% versus 97.5% in the first experiment and 97.8% versus 80.7% in the second. In the third experiment, PPM achieved a lower accuracy rate than SMO with 53.2% versus 60.2%. Code-switching between Egyptian Arabic and English text is easiest to detect because Arabic and English are generally written in different character sets. It is more difficult to distinguish between Arabic dialects and MSA as these use the same character set, and most users of Arabic, especially Saudis and Egyptians, frequently mix MSA with their dialects. We also note that the MSA corpus used for training the MSA model may not represent MSA Facebook text well, being built from news websites. This paper also describes in detail the new Arabic corpora created for this research and our experiments.

show abstract

“…The secondary language is usually manifested in the form of short expressions such as words and phrases [8]. According to [9] yung ice.")…”

Section: A Code-switching Point Detectionmentioning

confidence: 99%

“…There have been a lot of researches regarding code-switching. These researches focus mainly on topics such as speech recognition, language identification, text-to-speech synthesis and code-switching speech database creation [8].…”

Section: A Code-switching Point Detectionmentioning

confidence: 99%

Detection of Intra-Sentential Code-Switching Points Using Word Bigram and Unigram Frequency Count

Clarisse¹,

Bacatan²,

Loren³

et al. 2014

IJCCE

View full text Add to dashboard Cite

Abstract-Detecting code-switching points is important, especially with the increasing globalism and multilingualism. However, this is a challenging task, but with the help of computers and technology, this can be done easily. In this paper, an approach to effectively detect code-switching points in a Tagalog-English text input, especially those with alternating English and Tagalog words, is presented. The approach uses the frequency counts of word bigrams and unigrams from language models which were trained from an existing and available corpus. For the testing, 3 test data categories were used -twitter posts, conversations, and short stories. The test data were composed of a total of 3088 English and Tagalog words. The results show that the system's accuracy of properly identifying English and Tagalog words ranged from 81% -95%, while the F-measure ranged from 72% -95%. The research can be extended and improved using other n-grams, stemming, and searching algorithms. IndexTerms-Code-switching point detection, intra-sentential code-switching, word bigram, word unigram.

show abstract

Identification of Code-Switched Sentences and Words Using Language Modeling Approaches

Cited by 7 publications

References 23 publications

Subsegmental language detection in Celtic language text

Subsegmental language detection in Celtic language text

Compression versus traditional machine learning classifiers to detect code-switching in varieties and dialects: Arabic as a case study

Detection of Intra-Sentential Code-Switching Points Using Word Bigram and Unigram Frequency Count

Contact Info

Product

Resources

About