Reinforced Iterative Knowledge Distillation for Cross-Lingual Named Entity Recognition

Abstract

Named entity recognition (NER) is a fundamental component in many applications, such as Web Search and Voice Assistants. Although deep neural networks greatly improve the performance of NER, due to the requirement of large amounts of training data, deep neural networks can hardly scale out to many languages in an industry setting. To tackle this challenge, cross-lingual NER transfers knowledge from a rich-resource language to languages with low resources through pre-trained multilingual language models. Instead of using training data in target languages, cross-lingual NER has to rely on only training data in source languages, and optionally adds the translated training data derived from source languages. However, the existing cross-lingual NER methods do not make good use of rich unlabeled data in target languages, which is relatively easy to collect in industry applications. To address the opportunities and challenges, in this paper we describe our novel practice in Microsoft to leverage such large amounts of unlabeled data in target languages in real production settings. To effectively extract weak supervision signals from the unlabeled data, we develop a novel approach based on the ideas of semi-supervised learning and reinforcement learning. The empirical study on three benchmark data sets verifies that our approach establishes the new state-ofthe-art performance with clear edges. Now, the NER techniques reported in this paper are on their way to become a fundamental component for Web ranking, Entity Pane, Answers Triggering, and Question Answering in the Microsoft Bing search engine. Moreover, our techniques will also serve as part of the Spoken Language Understanding module for a commercial voice assistant. We plan to open source the code of the prototype framework after deployment.

DA-Net: A Disentangled and Adaptive Network for Multi-Source Cross-Lingual Transfer Learning

“…(2) Feature-based approaches rely on features alignment to diminish the language shift (Chen et al 2021;Ge et al 2023). (3) The distillation-based methods (Wu et al 2020b;Liang et al 2021;Ma et al 2022) enable the student network to gain task knowledge from soft labels predicted by the teacher network on the target language. However, these studies are designed on the single-source assumption and fail to deal with multiple source languages.…”

Bridging the Gap between Language Models and Cross-Lingual Sequence Labeling

“…Recently, there is a growing body of literature that recognizes the importance of cross-lingual sequence labeling tasks (xSL for short) (Huang et al, 2019;Lewis et al, 2020;Artetxe et al, 2019a), where translation systems are utilized to translate high-resource languages into low-resource languages so as to enrich the training data. However, the performance of xSL models is severely affected by translation quality (Yuan et al, 2020;Liang et al, 2021a).…”

“…However, when pushing the boundary of SL to low-resource languages, xSL can be very challenging, due to limited training data. To tackle this challenge, various efforts have been proposed (Huang et al, 2019;Liang et al, 2021a). Cui et al (2019) and Singh et al (2019) utilized machine translation to get parallel data as the data argumentation method.…”

FreeKD: Free-direction Knowledge Distillation for Graph Neural Networks

“…Following [34], our actions are sampled in batch, and obtain the delayed reward after two GNNs being updated according to a batch of sequential actions. Similar to [21], we utilize the performance of the models after being updated as the reward. We use the negative value of the cross entropy loss to measure the performance of the models as in [21,42], defined as:…”

“…Similar to [21], we utilize the performance of the models after being updated as the reward. We use the negative value of the cross entropy loss to measure the performance of the models as in [21,42], defined as:…”