An Empirical Study of Extrapolation in Text Generation with Scalar Control

Abstract

We conduct an empirical evaluation of extrapolation performance when conditioning on scalar control inputs like desired output length, desired edit from an input sentence, and desired sentiment across three text generation tasks. Specifically, we examine a zero-shot setting where models are asked to generalize to ranges of control values not seen during training. We focus on evaluating popular embedding methods for scalar inputs, including both learnable and sinusoidal embeddings, as well as simpler approaches. Surprisingly, our findings indicate that the simplest strategy of using scalar inputs directly, without further encoding, most reliably allows for successful extrapolation. Related WorkKikuchi et al. (2016) and Liu et al. (2018) modified the initial state of the decoder in seq2seq models in order to condition on control values. Sennrich et al.