Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model

Hsu, Tsung-Yuan; Liu, Chi-Liang; Lee, Hung-yi

doi:10.18653/v1/d19-1607

Cited by 40 publications

(40 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zero shot cross lingual transfer learning (Conneau et al, 2018;Pires et al, 2019) in the NLP context refers to transferring a model which is trained to solve a specific task in a source language to solve that specific task in a different language. Initially we get our baseline results using the zero-shot setting, where we use pretrained transformer models fine-tuned on English question answering task and check their performance on the Bengali evaluation dataset following similar research on Chinese (Hsu et al, 2019), French (d'Hoffschmidt et al, 2020 and Japanese (Siblini et al, 2019) language. In the next section we fine-tune these models further with our translated Bengali SQuAD dataset and compare the baselines with the fine-tuned models.…”

Section: Experiments and Resultsmentioning

confidence: 99%

“…Aside from using translated dataset, there has also been attempts of curating large question answering datasets in multiple other languages including French (d'Hoffschmidt et al, 2020), Korean (Lim et al, 2019), Russian (Efimov et al, 2020), Chinese (Cui et al, 2018;Shao et al, 2018) and benchmark models like QANet (Yu et al, 2018), BiDAF (Seo et al, 2016), BERT (Devlin et al, 2018) have been trained on them. In contrast to gathering translated or human annotated dataset for model training, zero shot transfer learning where pretrained models were evaluated directly on a new language after task specific training on question answering has also been attempted on reading comprehension tasks (Artetxe et al, 2019;Hsu et al, 2019;Siblini et al, 2019). To the best of our knowledge none of similar work have ever been attempted on Bengali so far.…”

Section: Question Answering In Englishmentioning

confidence: 99%

“…The reason that this process is efficient is because the model has the ability to perform extractive question answering, but not necessarily in the desired language. Existing research (Hsu et al, 2019) on cross-lingual transfer learning on reading comprehension datasets showed successful results using multilingual BERT for zero shot reading comprehension. The authors found that multi-BERT fine-tuned on training set in source language and evaluated on a target language showed comparable performance to models trained from scratch and that m-BERT model has the ability to transfer between low lexical similarity language pairs such as English and Chinese.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Deep learning based question answering system in Bengali

Mayeesha

Sarwar

Rahman

2020

Journal of Information and Telecommunication

View full text Add to dashboard Cite

Recent advances in the field of natural language processing has improved state-of-the-art performances on many tasks including question answering for languages like English. Bengali language is ranked seventh and is spoken by about 300 million people all over the world. But due to lack of data and active research on QA similar progress has not been achieved for Bengali. Unlike English, there is no benchmark large scale QA dataset collected for Bengali, no pretrained language model that can be modified for Bengali question answering and no human baseline score for QA has been established either. In this work we use state-of-theart transformer models to train QA system on a synthetic reading comprehension dataset translated from one of the most popular benchmark datasets in English called SQuAD 2.0. We collect a smaller human annotated QA dataset from Bengali Wikipedia with popular topics from Bangladeshi culture for evaluating our models. Finally, we compare our models with human children to set up a benchmark score using survey experiments.

show abstract

Section: Experiments and Resultsmentioning

confidence: 99%

Section: Question Answering In Englishmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deep learning based question answering system in Bengali

Mayeesha

Sarwar

Rahman

2020

Journal of Information and Telecommunication

View full text Add to dashboard Cite

show abstract

“…For this reason, research has been focused lately on models that can work in a zero-shot setting, i.e., without being explicitly trained on data from the target language or domain. This training paradigm has been utilized with great effect for several popular NLP problems, such as cross-lingual document retrieval [25], sequence labeling [26], cross-lingual dependency parsing [27], and reading comprehension [28]. More specific to classification tasks, Ye et al [29] developed a reinforcement learning framework for cross-task text classification, which was tested also on the problem of sentiment classification in a monolingual setting.…”

Section: Related Workmentioning

confidence: 99%

Zero-Shot Learning for Cross-Lingual News Sentiment Classification

et al. 2020

View full text Add to dashboard Cite

In this paper, we address the task of zero-shot cross-lingual news sentiment classification. Given the annotated dataset of positive, neutral, and negative news in Slovene, the aim is to develop a news classification system that assigns the sentiment category not only to Slovene news, but to news in another language without any training data required. Our system is based on the multilingual BERTmodel, while we test different approaches for handling long documents and propose a novel technique for sentiment enrichment of the BERT model as an intermediate training step. With the proposed approach, we achieve state-of-the-art performance on the sentiment analysis task on Slovenian news. We evaluate the zero-shot cross-lingual capabilities of our system on a novel news sentiment test set in Croatian. The results show that the cross-lingual approach also largely outperforms the majority classifier, as well as all settings without sentiment enrichment in pre-training.

show abstract

“…mBERT is a multilingual version of BERT, which is trained on Wikipedia monolingual corpora in 104 languages. This model proves to be surprisingly effective in a wide range of cross-lingual tasks [32] [33], e.g., reading comprehension, document classification, etc.…”

Section: Baselinementioning

confidence: 99%

Cross-Lingual Passage Re-Ranking With Alignment Augmented Multilingual BERT

Chen

Zhang

et al. 2020

IEEE Access

View full text Add to dashboard Cite

The task of Cross-lingual Passage Re-ranking (XPR) aims to rank a list of candidate passages in multiple languages given a query, which is generally challenged by two main issues: (1) the query and passages to be ranked are often in different languages, which requires strong cross-lingual alignment, and (2) the lack of annotated data for model training and evaluation. In this article, we propose a two-stage approach to address these issues. At the first stage, we introduce the task of Cross-lingual Paraphrase Identification (XPI) as an extra pre-training to augment the alignment by leveraging a large unsupervised parallel corpus. This task aims to identify whether two sentences, which may be from different languages, have the same meaning. At the second stage, we introduce and compare three effective strategies for cross-lingual training. To verify the effectiveness of our method, we construct an XPR dataset by assembling and modifying two monolingual datasets. Experimental results show that our augmented pre-training contributes significantly to the XPR task. Besides, we directly transfer the trained model to test on out-domain data which are constructed by modifying three multilingual Question Answering (QA) datasets. The results demonstrate the cross-domain robustness of the proposed approach.

show abstract

Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model

Cited by 40 publications

References 17 publications

Deep learning based question answering system in Bengali

Deep learning based question answering system in Bengali

Zero-Shot Learning for Cross-Lingual News Sentiment Classification

Cross-Lingual Passage Re-Ranking With Alignment Augmented Multilingual BERT

Contact Info

Product

Resources

About