Code-Mixed to Monolingual Translation Framework

Mahata, Sainik Kumar; Mandal, Soumil; Das, Dipankar; Bandyopadhyay, Sivaji

doi:10.1145/3368567.3368579

Cited by 11 publications

(4 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As for work on CS MT, there are many efforts (Sinha and Thakur, 2005;Dhar et al, 2018;Mahata et al, 2019;Menacer et al, 2019;Song et al, 2019;Tarunesh et al, 2021;Xu and Yvon, 2021;Chen et al, 2022;Hamed et al, 2022c). To the best of our knowledge, none of these efforts presented an extensive comparison covering different segmentation techniques.…”

Section: Related Workmentioning

confidence: 99%

Exploring Segmentation Approaches for Neural Machine Translation of Code-Switched Egyptian Arabic-English Text

Gaser,

Mager,

Hamed

et al. 2023

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics

View full text Add to dashboard Cite

Data sparsity is one of the main challenges posed by code-switching (CS), which is further exacerbated in the case of morphologically rich languages. For the task of machine translation (MT), morphological segmentation has proven successful in alleviating data sparsity in monolingual contexts; however, it has not been investigated for CS settings. In this paper, we study the effectiveness of different segmentation approaches on MT performance, covering morphology-based and frequency-based segmentation techniques. We experiment on MT from code-switched Arabic-English to English. We provide detailed analysis, examining a variety of conditions, such as data size and sentences with different degrees of CS. Empirical results show that morphology-aware segmenters perform the best in segmentation tasks but under-perform in MT. Nevertheless, we find that the choice of the segmentation setup to use for MT is highly dependent on the data size. For extreme low-resource scenarios, a combination of frequency and morphology-based segmentations is shown to perform the best. For more resourced settings, such a combination does not bring significant improvements over the use of frequency-based segmentation.

show abstract

Section: Related Workmentioning

confidence: 99%

Exploring Segmentation Approaches for Neural Machine Translation of Code-Switched Egyptian Arabic-English Text

Gaser,

Mager,

Hamed

et al. 2023

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…Menacer et al [9] used this approach to translate Arabic-English code-mixed text into English. Mahata et al [15] used deep learning for Bengali-English code-mixed normalisation, employing a character-level long-short term memory network to perform language identification. Next, the English text was translated using a neural MT system, whilst the Bengali text was transliterated back to its Devanagri form.…”

Section: A Code-mixed Text Normalisationmentioning

confidence: 99%

“…The translation module translates codemixed text to monolingual text -here, Indonesian-English code-mixed text to Indonesian-language text. We slightly modified the MLF approach used by Barik et al by grouping neighbouring tokens with the same language in the text as one language segment; thus the translation is not performed for each token, but for each segment, similar to the approach taken in [15]. Because Barik et al separately applied translation to individual tokens, it potentially produces incorrect translation results since it prevents the MT system to know the correct context of the token.…”

Section: ) Translationmentioning

confidence: 99%

Normalisation of Indonesian-English Code-Mixed Text and its Effect on Emotion Classification

Yulianti¹,

Kurnia²,

Adriani³

et al. 2021

IJACSA

View full text Add to dashboard Cite

Usage of code-mixed text has increased in recent years among Indonesian internet users, who often mix Indonesian-language with English-language text. Normalisation of this code-mixed text into Indonesian needs to be performed to capture the meaning of English parts of the text and process them effectively. We improve a state-of-the-art code-mixed Indonesian-English normalisation system by modifying its pipeline modules. We further analyse the effect of code-mixed normalisation on emotion classification tasks. Our approach significantly improved on a state-of-the-art Indonesian-English code-mixed text normalisation system in both the individual pipeline modules and the overall system. The new feature set in the language identification module showed an improvement of 4.26% in terms of F 1 score. The combination of machine translation and ruleset in the lexical normalisation module improved BLEU score by 25.22% and lowered WER by 62.49%. The use of context in the translation module improved BLEU score by 2.5% and lowered WER by 8.84%. The effectiveness of the overall pipeline normalisation system increased by 32.11% and 33.82%, in terms of BLEU score and WER, respectively. Code-mixed normalisation also improved the accuracy of emotion classification by up to 37.74% in terms of F 1 score.

show abstract

“…LID is critical for some subsequent natural language processing tasks in code-mixed documents ( Gundapu & Mamidi, 2018 ). Applying LID in the code-mixed text has become a foundation work of various NLP systems, including sentiment analysis ( Ansari & Govilkar, 2018 ; Mahata, Das & Bandyopadhyay, 2021 ), translation ( Barik, Mahendra & Adriani, 2019 ; Mahata et al, 2019 ), and emotion classification ( Yulianti et al, 2021 ). The absence of LID in pre-processing tasks can affect those NLP systems.…”

Section: Introductionmentioning

confidence: 99%

Corpus creation and language identification for code-mixed Indonesian-Javanese-English Tweets

Hidayatullah

Apong

Lai

et al. 2023

PeerJ Computer Science

View full text Add to dashboard Cite

With the massive use of social media today, mixing between languages in social media text is prevalent. In linguistics, the phenomenon of mixing languages is known as code-mixing. The prevalence of code-mixing exposes various concerns and challenges in natural language processing (NLP), including language identification (LID) tasks. This study presents a word-level language identification model for code-mixed Indonesian, Javanese, and English tweets. First, we introduce a code-mixed corpus for Indonesian-Javanese-English language identification (IJELID). To ensure reliable dataset annotation, we provide full details of the data collection and annotation standards construction procedures. Some challenges encountered during corpus creation are also discussed in this paper. Then, we investigate several strategies for developing code-mixed language identification models, such as fine-tuning BERT, BLSTM-based, and CRF. Our results show that fine-tuned IndoBERTweet models can identify languages better than the other techniques. This is the result of BERT’s ability to understand each word’s context from the given text sequence. Finally, we show that sub-word language representation in BERT models can provide a reliable model for identifying languages in code-mixed texts.

show abstract

Code-Mixed to Monolingual Translation Framework

Cited by 11 publications

References 9 publications

Exploring Segmentation Approaches for Neural Machine Translation of Code-Switched Egyptian Arabic-English Text

Exploring Segmentation Approaches for Neural Machine Translation of Code-Switched Egyptian Arabic-English Text

Normalisation of Indonesian-English Code-Mixed Text and its Effect on Emotion Classification

Corpus creation and language identification for code-mixed Indonesian-Javanese-English Tweets

Contact Info

Product

Resources

About