Learning to Selectively Transfer

Abstract

Deep text matching approaches have been widely studied for many applications including question answering and information retrieval systems. To deal with a domain that has insufficient labeled data, these approaches can be used in a Transfer Learning (TL) setting to leverage labeled data from a resource-rich source domain. To achieve better performance, source domain data selection is essential in this process to prevent the "negative transfer" problem. However, the emerging deep transfer models do not fit well with most existing data selection methods, because the data selection policy and the transfer learning model are not jointly trained, leading to sub-optimal training efficiency.In this paper, we propose a novel reinforced data selector to select high-quality source domain data to help the TL model. Specifically, the data selector "acts" on the source domain data to find a subset for optimization of the TL model, and the performance of the TL model can provide "rewards" in turn to update the selector. We build the reinforced data selector based on the actor-critic framework and integrate it to a DNN based transfer learning model, resulting in a Reinforced Transfer Learning (RTL) method. We perform a thorough experimental evaluation on two major tasks for text matching, namely, paraphrase identification and natural language inference. Experimental results show the proposed RTL can significantly improve the performance of the TL model. We further investigate different settings of states, rewards, and policy optimization methods to examine the robustness of our method. Last, we conduct a case study on the selected data and find our method is able to select source domain data whose Wasserstein distance is close to the target domain data. This is reasonable and intuitive as * Work done during an internship at Alibaba Group.

Extracting personal information from conversations

“…Reinforcement learning (RL) methods are often applied in conversational models for response generation [63], based on the feedback from human quality assessment scores for the output utterances. Other NLP tasks where reinforcement learning can be applied include question answering [92,127], text classification [184] and entity linking [35].…”

Probabilistic Robustness for Data Filtering

“…Existing studies on data selection and robust learning demonstrate a need for test domain knowledge during training. Some data selection work (Moore and Lewis, 2010;Kirchhoff and Bilmes, 2014;van der Wees et al, 2017;Fan et al, 2017;Qu et al, 2019;Kang et al, 2020) chooses critical in-domain data for domain adaptation, and other work defends against adversarial attacks but offers little help for out-of-domain robustness (Taori et al, 2020) under natural distributional shifts (Wang et al, 2021) that occurs more frequently than extreme adversarial cases. This outof-domain robustness is often measured by testing on a specific domain and a single task like sentiment classification (Müller et al, 2019;Hendrycks et al, 2020).…”