Creating Multilingual Parallel Corpora in Indian Languages

Choudhary, Narayan; Jha, Girish Nath

doi:10.1007/978-3-319-14120-6_43

Cited by 9 publications

(10 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Along with harmonizing the chunks, this module marks the heads of each chunk in both languages using generalized rules defined by Sharma et al (2006). For clarity, we have mapped the POS tags from Penn Treebank POS tagsets (Marcus et al, 1993) for English and Bureau Of Indian Standard (BIS) POS tagset (Choudhary and Jha, 2011) for Bengali to the Universal Dependency Tagset . The second module in the pipeline facilitates rule-based chunk replacement by taking the chunkharmonized parallel Bengali and English sentences as inputs and replacing some selected Bengali chunks with English according to the rules discussed in 2.2.…”

Section: The Code-mixing Processmentioning

confidence: 99%

Proceedings of the 18th International Workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2019)

2019

View full text Add to dashboard Cite

The 18th edition of the International Workshop on Treebanks and Linguistics (TLT) follows an annual series that started in 2002, in Sozopol, Bulgaria. TLT addresses all aspects of treebank design, development, and use, "treebank" taken in a broad sense of any spoken or written data augmented with computationally processable annotations of linguistic structure at various levels. This year's edition is special as TLT is part of the first SyntaxFest, a grouping of four events, which took place in Paris, France, during the last week of August:

show abstract

Section: The Code-mixing Processmentioning

confidence: 99%

Proceedings of the 18th International Workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2019)

2019

View full text Add to dashboard Cite

show abstract

“…Along with harmonizing the chunks, this module marks the heads of each chunk in both languages using generalized rules defined by Sharma et al (2006). For clarity, we have mapped the POS tags from Penn Treebank POS tagsets (Marcus et al, 1993) for English and Bureau Of Indian Standard (BIS) POS tagset (Choudhary and Jha, 2011) for Bengali to the Universal Dependency Tagset (Nivre et al, 2016). The second module in the pipeline facilitates rule-based chunk replacement by taking the chunkharmonized parallel Bengali and English sentences as inputs and replacing some selected Bengali chunks with English according to the rules discussed in 2.2.…”

Section: The Code-mixing Processmentioning

confidence: 99%

Section: The Code-mixing Processmentioning

confidence: 99%

Dependency Parser for Bengali-English Code-Mixed Data enhanced with a Synthetic Treebank

Ghosh¹,

Sharma²,

Khanuja³

2019

Proceedings of the 18th International Workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2019)

View full text Add to dashboard Cite

The development of code-mixing (CM) NLP systems has significantly gained importance in recent times due to an upsurge in the usage of CM data by multilingual speakers. However, this proves to be a challenging task due to the complexities created by the presence of multiple languages together. The complexities get further compounded by the inconsistencies present in the raw data on social media and other platforms. In this paper, we present a neural stack based dependency parser for CM data of Bengali and English by utilizing pre-existing resources for closely related Hindi and English CM treebank as well as monolingual treebanks for Bengali, Hindi and English. To address the issue of scarcity of annotated resources for Bengali-English CM pair, we present a rule based system to computationally generate a synthetic code-mixing treebank for Bengali and English (Syn-BE) which is used to further improve the accuracy of our dependency parser. For evaluation purpose, we present a dataset of 500 Bengali-English tweets annotated under Universal Dependencies scheme.

show abstract

“…The parallel corpora for 4 Indian languages namely Hindi (hn), Marathi (mt), Gujarati (gj) and Bangla (bn) was taken from Indian Languages Corpora Initiative (ILCI) (Choudhary and Jha, 2011) . The parallel corpus used in our experiments belonged to two domains -health and tourism and the training set consisted of 28000 sentences.…”

Section: Baseline Translation Modelmentioning

confidence: 99%

“…In spite of several initiatives taken by numerous organizations to generate parallel corpora for different language pairs, training data for many language pairs is either not yet available or is insufficient for producing good SMT systems. Indian Languages Corpora Initiative (ILCI) (Choudhary and Jha, 2011) is currently the only reliable source for multilingual parallel corpora for Indian languages however the number of parallel sentences is still not sufficient to create high quality SMT systems.…”

Section: Introductionmentioning

confidence: 99%

Exploring System Combination approaches for Indo-Aryan MT Systems

Singla¹,

Singh²,

Shastri³

et al. 2014

Proceedings of the EMNLP'2014 Workshop on Language Technology for Closely Related Languages and Language Variants

View full text Add to dashboard Cite

Statistical Machine Translation (SMT) systems are heavily dependent on the quality of parallel corpora used to train translation models. Translation quality between certain Indian languages is often poor due to the lack of training data of good quality. We used triangulation as a technique to improve the quality of translations in cases where the direct translation model did not perform satisfactorily. Triangulation uses a third language as a pivot between the source and target languages to achieve an improved and more efficient translation model in most cases. We also combined multi-pivot models using linear mixture and obtained significant improvement in BLEU scores compared to the direct source-target models.

show abstract

Creating Multilingual Parallel Corpora in Indian Languages

Cited by 9 publications

References 4 publications

Proceedings of the 18th International Workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2019)

Proceedings of the 18th International Workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2019)

Dependency Parser for Bengali-English Code-Mixed Data enhanced with a Synthetic Treebank

Exploring System Combination approaches for Indo-Aryan MT Systems

Contact Info

Product

Resources

About