Sequence Prediction with Unlabeled Data by Reward Function Learning

Abstract

Reinforcement learning (RL), which has been successfully applied to sequence prediction, introduces reward as sequence-level supervision signal to evaluate the quality of a generated sequence. Existing RL approaches use the ground-truth sequence to define reward, which limits the application of RL techniques to labeled data. Since labeled data is usually scarce and/or costly to collect, it is desirable to leverage large-scale unlabeled data. In this paper, we extend existing RL methods for sequence prediction to exploit unlabeled data. We propose to learn the reward function from labeled data and use the predicted reward as pseudo reward for unlabeled data so that we can learn from unlabeled data using the pseudo reward. To get good pseudo reward on unlabeled data, we propose a RNN-based reward network with attention mechanism, trained with purposely biased data distribution. Experiments show that the pseudo reward can provide good supervision and guide the learning process on unlabeled data. We observe significant improvements on both neural machine translation and text summarization.

Exploiting Monolingual Data at Scale for Neural Machine Translation

“…Zhang and Zong (2016) propose self learning approach to generate the synthetic data for the source-side monolingual data, which is a semi-supervised method. Wu et al (2017) leverage the source-side monolingual data to train the NMT system by learning reward func-4200 tion in a reinforcement learning framework.…”

A Study of Reinforcement Learning for Neural Machine Translation

“…To address the inconsistency issue, reinforcement learning (RL) methods have been adopted to optimize sequence-level objectives. For example, policy optimization methods such as REINFORCE (Ranzato et al, 2016;Wu et al, 2017b) and actorcritic (Bahdanau et al, 2017) are leveraged for sequence generation tasks including NMT. In machine translation community, a similar method is proposed with the name 'minimum risk training' (Shen et al, 2016).…”

Look Harder: A Neural Machine Translation Model with Hard Attention

“…Even though RL based models are difficult to train, in recent years, multiple works (Mnih et al, 2014;Choi et al, 2017;Yu et al, 2017;Narayan et al, 2018;Sathish et al, 2018;Shen et al, 2018) have shown to improve the performance of several natural language processing tasks. Also, it has been used in NMT Wu et al, 2017;Bahdanau et al, 2017) to overcome the inconsistency between the token level objective function and sequence-level evaluation metrics such as BLEU. Our approach is also related to the method proposed by Lei et al (2016) to explain the decision of text classifier.…”