COD3S: Diverse Generation with Discrete Semantic Signatures

Abstract

We present COD3S, a novel method for generating semantically diverse sentences using neural sequence-to-sequence (seq2seq) models. Conditioned on an input, seq2seq models typically produce semantically and syntactically homogeneous sets of sentences and thus perform poorly on one-to-many sequence generation tasks. Our two-stage approach improves output diversity by conditioning generation on locality-sensitive hash (LSH)-based semantic sentence codes whose Hamming distances highly correlate with human judgments of semantic textual similarity. Though it is generally applicable, we apply COD3S to causal generation, the task of predicting a proposition's plausible causes or effects. We demonstrate through automatic and human evaluation that responses produced using our method exhibit improved diversity without degrading task performance.

InFillmore: Frame-Guided Language Generation with Bidirectional Context

“…Without the ability to handle explicit frame semantic guidance, this task would be incredibly difficult for a neural generation model. Weir et al (2020) explore the task of diverse causal generation, in which a model must propose a set of semantically distinct causes or effects of an input sentence. Following their two-step approach, we devise a frame semantic model that 1) predicts the distinct frames that are likely to appear at a specified index before (for causes) or after (effects) the input sentence, then 2) run a separate beam search conditioned on each top-k predicted A. depicts human-in-the-loop iterative story refinement, in which a user provides an initial context and/or intended frame semantic content and interacts with the model to predict and user-select new frame content and surface-realized context.…”

“…PPLM (Dathathri et al, 2020) makes use of lightweight attribute classifiers that guide generation without requiring language model retraining. For diverse generation of sentences in a more general scenario, Weir et al (2020) train models to condition on semantic bit codes obtained from hashing sentence embeddings.…”

Determinantal Beam Search

“…There are a number of works that address the problem of diverse decoding in language generation. Weir et al (2020) Table 2: We report the coverage (as defined in Eq. ( 14)) of 1, 2, 3, 4-grams and averaged across 1, 2, 3, 4-grams as well as median BLEU for k = 20 on the newstest dataset.…”

Toward Diverse Precondition Generation

“…Bao et al (2020) used K categorical latent variables to control the generation context of dialogue responses and pick the highest probability response from the responses generated using the latent variables. COD3S (Weir et al, 2020) is designed to generate diverse causal relations. It uses locality-sensitive hashing (LSH) (Indyk and Motwani, 1998) on representations from Sentence-BERT (Reimers and Gurevych, 2019).…”

“…To this end, in addition to the full target event description x, we also provide a precondition trigger marked by a special token -<E> precondition_event -at the end of the input. This can be seen as a form of a control code similar to those used in Keskar et al ( 2019); Weir et al (2020). The crucial difference, however, is that the codes in our case are dynamically generated conditioned on the input and not restricted to a predefined set.…”