Generationary or “How We Went beyond Word Sense Inventories and Learned to Gloss”

Abstract

Mainstream computational lexical semantics embraces the assumption that word senses can be represented as discrete items of a predefined inventory. In this paper we show this needs not be the case, and propose a unified model that is able to produce contextually appropriate definitions. In our model, Generationary, we employ a novel span-based encoding scheme which we use to fine-tune an English pre-trained Encoder-Decoder system to generate glosses. We show that, even though we drop the need of choosing from a predefined sense inventory, our model can be employed effectively: not only does Generationary outperform previous approaches in the generative task of Definition Modeling in many settings, but it also matches or surpasses the state of the art in discriminative tasks such as Word Sense Disambiguation and Word-in-Context. Finally, we show that Generationary benefits from training on data from multiple inventories, with strong gains on various zeroshot benchmarks, including a novel dataset of definitions for free adjective-noun phrases. The software and reproduction materials are available at http://generationary.org.

Definition Modelling for Appropriate Specificity

“…They used the average of sentence BLEU with single-reference based on Gadetsky et al (2018). The comparison of the obtained result with Bevilacqua et al (2020) is shown in Table 10. They used the corpus BLEU calculated by sacreBLEU script (Post, 2018).…”

“…Therefore, these previous studies often result in OOV definitions, i.e., 'a target is unk ,' particularly for non-standard languages (e.g., internet slang). Bevilacqua et al (2020) employed the pretrained BART (Lewis et al, 2020) for definition generation to address the problem. Furthermore, these studies do not have any mechanism to consider the specificity of the generated definitions.…”

“…We further present experimental results for relevant comparisons withWashio et al (2019),Bevilacqua et al (2020) in the appendices.…”

ProtAugment: Intent Detection Meta-Learning through Unsupervised Diverse Paraphrasing

“…The BART (Lewis et al, 2020) model is a Transformer-based neural machine translation architecture that is trained to remove artificially corrupted text from the input thanks to an autoencoder architecture. While it is trained to reconstruct the original noised input, it can be fine-tuned for taskspecific conditional generation by minimizing the cross-entropy loss on new training input-output pairs (Bevilacqua et al, 2020). In PROTAUGMENT, we fine-tune a pre-trained BART model on the paraphrasing task.…”

“…Increasing the number of epochs for fine-tuning BART degrades performances on the intent detection task: the downstream diverse beam search struggles to find diverse enough beam groups since the model perplexity has been lower with further fine-tuning (this is also hinted in (Bevilacqua et al, 2020)). Our text encoder f φ is a bert-base model, and the embedding of a given sentence is the last layer hidden state of the first token of this sentence.…”