Language Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic Data

Pratapa, Adithya; Bhat, Gayatri; Choudhury, Monojit; Sitaram, Sunayana; Dandapat, Sandipan; Bali, Kalika

doi:10.18653/v1/p18-1143

Cited by 99 publications

(113 citation statements)

References 31 publications

Supporting

Mentioning

110

Contrasting

Order By: Relevance

“…In a recent work (Pratapa et al, 2018), we presented a methodology to generate linguistic theory based synthetic CM data (gCM) and showed its effectiveness in CM language modeling. Synthetic CM data was generated by employing Equivalence Constraint (EC) theory (Poplack, 1980;Sankoff, 1998).…”

Section: Bilingual Skip-gram Model (Biskip)mentioning

confidence: 99%

“…In this paper, we compare three popular bilingual word embedding techniques (Sec 2): Bilingual correlation based embeddings (BiCCA) (Faruqui and Dyer, 2014), Bilingual compositional model (BiCVM) (Hermann and Blunsom, 2014) and Bilingual Skip-gram (BiSkip) (Luong et al, 2015) on two tasks for CM text -sentiment analysis, a semantic task, and POS tagging, a syntactic task. On the same tasks, we also compare word embeddings learnt from synthetic CM data (generated using linguistic models as proposed in a recent work (Pratapa et al, 2018)) (Sec 3). Note that Wick et al (2016) use artificial code mixed data to learn multilingual embeddings for cross-lingual tasks, but their aim is to generate bilingual embeddings for monolingual or cross-lingual tasks.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Word Embeddings for Code-Mixed Language Processing

Pratapa¹,

Choudhury²,

Sitaram³

2018

Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

Self Cite

View full text Add to dashboard Cite

We compare three existing bilingual word embedding approaches, and a novel approach of training skip-grams on synthetic code-mixed text generated through linguistic models of code-mixing, on two tasks-sentiment analysis and POS tagging for code-mixed text. Our results show that while CVM and CCA based embeddings perform as well as the proposed embedding technique on semantic and syntactic tasks respectively, the proposed approach provides the best performance for both tasks overall. Thus, this study demonstrates that existing bilingual embedding techniques are not ideal for code-mixed text processing and there is a need for learning multilingual word embedding from the code-mixed text.

show abstract

Section: Bilingual Skip-gram Model (Biskip)mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Word Embeddings for Code-Mixed Language Processing

Pratapa¹,

Choudhury²,

Sitaram³

2018

Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

Self Cite

View full text Add to dashboard Cite

show abstract

“…Generating synthesized CS data using only monolingual data has been also explored in [11,12,13,14], however, they only address the textual data scarcity problem. Figure 1 illustrates the baseline E2E-CS-ASR model based on hybrid CTC/Attention architecture [15] which incorporates the advantages of both Connectionist Temporal Classification (CTC) model [16] and attention-based encoder-decoder model [17].…”

Section: Related Workmentioning

confidence: 99%

Constrained Output Embeddings for End-to-End Code-Switching Speech Recognition with Only Monolingual Data

Khassanov

Pham

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

The lack of code-switch training data is one of the major concerns in the development of end-to-end code-switching automatic speech recognition (ASR) models. In this work, we propose a method to train an improved end-to-end code-switching ASR using only monolingual data. Our method encourages the distributions of output token embeddings of monolingual languages to be similar, and hence, promotes the ASR model to easily code-switch between languages. Specifically, we propose to use Jensen-Shannon divergence and cosine distance based constraints. The former will enforce output embeddings of monolingual languages to possess similar distributions, while the later simply brings the centroids of two distributions to be close to each other. Experimental results demonstrate high effectiveness of the proposed method, yielding up to 4.5% absolute mixed error rate improvement on Mandarin-English codeswitching ASR task.

show abstract

“…Linguistic studies show that bilingual speakers switch languages by following various complex constraints [18,17] which may even include the intensity of sentiment expressed in various segments of text [23]. [20] synthesized code-mixed sentences by leveraging linguistic constraints arising from Equivalence Constraint Theory. While this works well for language pairs with good structural correspondence (like English-Spanish), we observe performance degrades with weaker correspondence (like English-Hindi).…”

Section: Introductionmentioning

confidence: 99%

A Deep Generative Model for Code Switched Text

Samanta

Reddy

Jagirdar

et al. 2019

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

Code-switching, the interleaving of two or more languages within a sentence or discourse is pervasive in multilingual societies. Accurate language models for code-switched text are critical for NLP tasks. Stateof-the-art data-intensive neural language models are difficult to train well from scarce language-labeled code-switched text. A potential solution is to use deep generative models to synthesize large volumes of realistic code-switched text. Although generative adversarial networks and variational autoencoders can synthesize plausible monolingual text from continuous latent space, they cannot adequately address codeswitched text, owing to their informal style and complex interplay between the constituent languages. We introduce VACS, a novel variational autoencoder architecture specifically tailored to code-switching phenomena. VACS encodes to and decodes from a two-level hierarchical representation, which models syntactic contextual signals in the lower level, and language switching signals in the upper layer. Sampling representations from the prior and decoding them produced well-formed, diverse code-switched sentences. Extensive experiments show that using synthetic code-switched text with natural monolingual data results in significant (33.06%) drop in perplexity.

show abstract

Language Modeling for Code-Mixing: The Role of Linguistic Theory based Synthetic Data

Cited by 99 publications

References 31 publications

Word Embeddings for Code-Mixed Language Processing

Word Embeddings for Code-Mixed Language Processing

Constrained Output Embeddings for End-to-End Code-Switching Speech Recognition with Only Monolingual Data

A Deep Generative Model for Code Switched Text

Contact Info

Product

Resources

About