Word-Level Language Identification and Predicting Codeswitching
            Points in Swahili-English Language Data

Piergallini, Mario; Shirvani, Rouzbeh A.; Gautam, Gauri Shankar; Chouikha, Mohamed F.

doi:10.18653/v1/w16-5803

Cited by 21 publications

(23 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Predicting C-S is important for modeling multilingual speech in NLP [19,20,21], in TTS [22,10], and in ASR [23,24,12,25,26,27,28,29]. Of particular importance for our work are findings from ASR that indicate that individual speakers or speakers from different nationalities show different patterns of Mandarin-English switching.…”

Section: Related Workmentioning

confidence: 99%

Metrics for Modeling Code-Switching Across Corpora

Guzmán¹,

Ricard²,

Serigos

et al. 2017

Interspeech 2017

View full text Add to dashboard Cite

In developing technologies for code-switched speech, it would be desirable to be able to predict how much language mixing might be expected in the signal and the regularity with which it might occur. In this work, we offer various metrics that allow for the classification and visualization of multilingual corpora according to the ratio of languages represented, the probability of switching between them, and the time-course of switching. Applying these metrics to corpora of different languages and genres, we find that they display distinct probabilities and periodicities of switching, information useful for speech processing of mixed-language data.

show abstract

Section: Related Workmentioning

confidence: 99%

Metrics for Modeling Code-Switching Across Corpora

Guzmán¹,

Ricard²,

Serigos

et al. 2017

Interspeech 2017

View full text Add to dashboard Cite

show abstract

“…However, for the SPA-ENG data set the system by (Shirvani et al, 2016) was the best performing at both the tweet and token level evaluations. On the other hand, the system by (Samih et al, 2016) was the best performing at both tweet and token level for the MSA-DA data set.…”

Section: Resultsmentioning

confidence: 93%

“…The best performing system here was (Shirvani et al, 2016) with an Avg-F-measure of 91.3%, which is Table 6: Tweet level performance results. We ranked the systems using the weighted average F-measure, Weighted-F1.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Proceedings of the Second Workshop on Computational Approaches to Code Switching

2016

View full text Add to dashboard Cite

ii Introduction Code-switching (CS) is the phenomenon by which multilingual speakers switch back and forth between their common languages in written or spoken communication. CS is pervasive in informal text communications such as news groups, tweets, blogs, and other social media of multilingual communities. Such genres are increasingly being studied as rich sources of social, commercial and political information. Apart from the informal genre challenge associated with such data within a single language processing scenario, the CS phenomenon adds another significant layer of complexity to the processing of the data. Efficiently and robustly processing CS data presents a new frontier for our NLP algorithms on all levels. The goal of this workshop is to bring together researchers interested in exploring these new frontiers, discussing state of the art research in CS, and identifying the next steps in this fascinating research area.The workshop program includes exciting papers discussing new approaches for CS data and the development of linguistic resources needed to process and study CS. We received a total of 12 regular workshop submissions of which we accepted nine for publication four of them as workshop talks and five as posters. The accepted workshop submissions cover a wide variety of language combinations from languages such as English, Hindi, Swahili, Mandarin, Dialectical Arabic and Modern Standard Arabic. The majority of the papers focus on social media data such as Twitter, and discussion fora.Another component of the workshop is the Second Shared Task on Language Identification of CS Data. The shared task focused on social media and included two language pairs: Modern Standard ArabicDialectal Arabic and English-Spanish. We received a total of 14 system runs from nine different teams. All teams except one submitted a shared task paper describing their system. All shared task systems will be presented during the workshop poster session and two of them will also present a talk. We would like to thank all authors who submitted their contributions to this workshop and all shared task participants for taking on the challenge of language identification in code switched data. We also thank the program committee members for their help in providing meaningful reviews. Lastly, we thank the EMNLP 2016 organizers for the opportunity to put together this workshop. AbstractThis paper addresses challenges of Natural Language Processing (NLP) on non-canonical multilingual data in which two or more languages are mixed. It refers to code-switching which has become more popular in our daily life and therefore obtains an increasing amount of attention from the research community. We report our experience that covers not only core NLP tasks such as normalisation, language identification, language modelling, part-of-speech tagging and dependency parsing but also more downstream ones such as machine translation and automatic speech recognition. We highlight and discuss the key problems for each of the tasks with supporting...

show abstract

“…These methods are language-dependent and require large annotated datasets or comprehensive dictionary of the target languages. For instance, some of the recent studies such as (Barman, Wagner, Vyas, Gella, Sharma, Bali, & Choudhury, 2014;Chrupala, & Foster, 2014;Dias Cardoso & Roy, 2016;Gella, Sharma, & Bali, 2013;Lavergne, Adda, Adda-Decker, & Lamel, 2014;Piergallini, Shirvani, Gautam, & Chouikha, 2016;Rijhwani, Sequiera, Choudhury, Bali, & Maddila, 2017;Barman, Das, Wagner, & Foster, 2014) used dictionary-based methods for LID at word level. While other studies such as (Banerjee et al, 2014(Banerjee et al, , 2014Chittaranjan, Vyas, Bali, & Choudhury, 2014;Dahiya, 2017;Das & Gambäck, 2014;Jaech, Mulcaire, Hathi, Ostendorf, & Smith, 2016;Jhamtani, Bhogi, & Raychoudhury, 2014;King & Abney, 2013;Mandal, Banerjee, Naskar, Rosso, & Bandyopadhyay, 2015;Nguyen & Dogruoz, 2013;Řehŭřek & Kolkus, 2009) used a combination of at least two of the following methods: dictionary-based methods, rule-based methods, character n-gram modelling and heuristics based on word level features modelling.…”

Section: Introductionmentioning

confidence: 99%

A word‐level language identification strategy for resource‐scarce languages

Asubiaro

Adegbola²,

Mercer

et al. 2018

Proc. Assoc. Info. Sci. Tech.

View full text Add to dashboard Cite

This study is based on the premise that it is possible to train computers to predict the language of a word (textual or audio) by learning from its character n‐gram pattern, without recourse to the language's dictionary. With the growth of multilingual collections and a need for automatic means of cleaning textual datasets, this paper presents a strategy for language identification of individual words in a body of texts. This strategy is suitable for resource‐scarce languages that do not have large electronic datasets that are required for machine learning and natural language processing studies and whose dictionaries may not be available. In this study, we focused on three African languages, namely Hausa, Igbo, and Yoruba. A training corpus in each of these languages was used to obtain the probabilities of character trigrams in the language. Given that English is a common language that is often mixed with these resource‐scarce languages in texts, we also obtained the probabilities of trigrams in an English training corpus. These probabilities were then used in identifying the language of each word in test corpora containing bilingual texts. Our strategy achieved average precision, recall and F1 values of about 97%, 91% and 94% respectively.

show abstract

Word-Level Language Identification and Predicting Codeswitching Points in Swahili-English Language Data

Cited by 21 publications

References 10 publications

Metrics for Modeling Code-Switching Across Corpora

Metrics for Modeling Code-Switching Across Corpora

Proceedings of the Second Workshop on Computational Approaches to Code Switching

A word‐level language identification strategy for resource‐scarce languages

Contact Info

Product

Resources

About