Bridging the Gap between Language Models and Cross-Lingual Sequence Labeling

Abstract

Large-scale cross-lingual pre-trained language models (xPLMs) have shown effectiveness in cross-lingual sequence labeling tasks (xSL), such as cross-lingual machine reading comprehension (xMRC) by transferring knowledge from a high-resource language to low-resource languages. Despite the great success, we draw an empirical observation that there is a training objective gap between pre-training and finetuning stages: e.g., mask language modeling objective requires local understanding of the masked token and the span-extraction objective requires global understanding and reasoning of the input passage/paragraph and question, leading to the discrepancy between pretraining and xMRC. In this paper, we first design a pre-training task tailored for xSL named Cross-lingual Language Informative Span Masking (CLISM) to eliminate the objective gap in a self-supervised manner. Second, we present ContrAstive-Consistency Regularization (CACR), which utilizes contrastive learning to encourage the consistency between representations of input parallel sequences via unsupervised cross-lingual instance-wise training signals during pre-training. By these means, our methods not only bridge the gap between pretrain-finetune, but also enhance PLMs to better capture the alignment between different languages. Extensive experiments prove that our method achieves clearly superior results on multiple xSL benchmarks with limited pretraining data. Our methods also surpass the previous state-of-the-art methods by a large margin in few-shot data settings, where only a few hundred training examples are available.

Orca: A Few-shot Benchmark for Chinese Conversational Machine Reading Comprehension

“…Furthermore, we collect Orca in few-shot settings, challenging models to learn unseen domains with few samples. Although one can question previous full-data CMRC datasets also could support few-shot training for CMRC models, we argue that it could lead to unclear comparisons due to the inconsistent settings of different works on these datasets (Chen et al, 2022b). In contrast, we present a single standard benchmark for thorough comparisons.…”

Exploring and Exploiting Multi-Granularity Representations for Machine Reading Comprehension

“…Based on these challenging datasets, a great number of end-to-end approaches have been proposed, including BiDAF (Seo et al, 2016), DCN (Xiong et al, 2016), R-Net (Wang et al, 2017). In MRC tasks, attention mechanism (Dong et al, 2020a;Gao et al, 2020;Zhu et al, 2020;Chen et al, 2022) have become an essential part to capture dependencies without considering their distance in the input/output sequences. Recently, some works show that well pre-trained models are powerful and convenient for downstream tasks, such as R-Trans (Liu et al, 2019a), DCMN+ , ALBERT (Lan et al, 2020) and GF-Net (Lee and Kim, 2020), which facilitate us to take pre-trained models as our backbone encoder.…”

Natural Response Generation for Chinese Reading Comprehension

“…Much of this popularity can be attributed to the release of many annotated and publicly available datasets (Rajpurkar et al, 2016;Trischler et al, 2016;Chen et al, 2022a;You et al, 2022;Chen et al, 2023a). Formally, these MRC efforts can be classified into two most popular streams 1 from the answer type perspective: span-extraction (Rajpurkar et al, 2016;Trischler et al, 2016;Cui et al, 2019;Chen et al, 2022b;You et al, 2021a) and multiple choices (Lai et al, 2017;Zellers et al, 2018;Wang et al, 2020). The former requires the model to locate the text span in the given passage as the answer, e.g., SQuAD (Rajpurkar et al, 2016) and NewQA (Trischler et al, 2016).…”