Improving Cross-lingual Text Classification with Zero-shot Instance-Weighting

Abstract

Cross-lingual text classification (CLTC) is a challenging task made even harder still due to the lack of labeled data in low-resource languages. In this paper, we propose zero-shot instance-weighting, a general model-agnostic zero-shot learning framework for improving CLTC by leveraging source instance weighting. It adds a module on top of pre-trained language models for similarity computation of instance weights, thus aligning each source instance to the target language. During training, the framework utilizes gradient descent that is weighted by instance weights to update parameters. We evaluate this framework over seven target languages on three fundamental tasks and show its effectiveness and extensibility, by improving on F1 score up to 4% in singlesource transfer and 8% in multi-source transfer. To the best of our knowledge, our method is the first to apply instance weighting in zeroshot CLTC. It is simple yet effective and easily extensible into multi-source transfer.

EHRKit: A Python Natural Language Processing Toolkit for Electronic Health Record Texts

“…In the future, we will investigate more EHR-NLP tasks including machine translation for more languages, multi-document summarization and question answering (Li et al, 2021b). Besides, we plan to investigate better-performed NLP models for these tasks, for example, BERTbased models (Lee et al, 2020;Li et al, 2021c) and graph-based models (Li et al, , 2021d.…”

A new amber outcrop from the Oligocene Nanning Basin of Guangxi, southern China

“…The decoder architecture, relying on the Transformer models, has found widespread application in various natural language generation tasks. Thus, following previous works (Li et al 2023), our method employs a lightweight Transformerbased decoder as the generation model. Both image features and auxiliary object features should be involved in decoding.…”

“…We follow Karpathy (Karpathy and Fei-Fei 2015) Evaluation Metrics. Following common settings (Tewel et al 2022;Li et al 2023), there are several metrics considered to evaluate the generated caption, including BLEU-4 (B@4) (Papineni et al 2002), METEOR (M) (Banerjee and Lavie 2005), ROUGE-L (R-L) (Lin 2004), CIDEr (C) (Vedantam, Lawrence Zitnick, and Parikh 2015), and SPICE (S) (Anderson et al 2016).…”

“…The presence of the modality gap (Liang et al 2022;Jiang et al 2023) poses a challenge in those text-only methods based on CLIP, as it hinders the direct use of visual embeddings as the source to generate captions. Towards this issue, some works propose solutions by noise injection or feature projection (Nukrai, Mokady, and Globerson 2022;Li et al 2023), assisting image features from CLIP to better align with text features during inference, while the modality gap still exists. Due to the absence of images, only textual information could be utilized for training while real images for inference.…”

“…During inference, they decode captions from the given image's features extracted by CLIP. In contrast to other unsupervised methods, text-only image captioning utilizes textual data during training without the accessibility to realworld images (Li et al 2023). Although previous methods have shown competitive results, they heavily depend on the CLIP's cross-domain capability.…”