AStitchInLanguageModels: Dataset and Methods for the Exploration of Idiomaticity in Pre-Trained Language Models

Abstract

Despite their success in a variety of NLP tasks, pre-trained language models, due to their heavy reliance on compositionality, fail in effectively capturing the meanings of multiword expressions (MWEs), especially idioms. Therefore, datasets and methods to improve the representation of MWEs are urgently needed. Existing datasets are limited to providing the degree of idiomaticity of expressions along with the literal and, where applicable, (a single) non-literal interpretation of MWEs. This work presents a novel dataset of naturally occurring sentences containing MWEs manually classified into a fine-grained set of meanings, spanning both English and Portuguese. We use this dataset in two tasks designed to test i) a language model's ability to detect idiom usage, and ii) the effectiveness of a language model in generating representations of sentences containing idioms. Our experiments demonstrate that, on the task of detecting idiomatic usage, these models perform reasonably well in the one-shot and few-shot scenarios, but that there is significant scope for improvement in the zero-shot scenario. On the task of representing idiomaticity, we find that pre-training is not always effective, while finetuning could provide a sample efficient method of learning representations of sentences containing MWEs.

Sample Efficient Approaches for Idiomaticity Detection

“…Table 3 presents the results of our best PET-based models alongside our BERTRAM-based model on the test set, as well as the mBERT system presented in (Tayyar Madabushi et al, 2022), for comparison. For each model we present the F1 macro score on the test set for each language, as well as the overall F1 macro score.…”

“…In evaluating the models presented in this work we use the Task 2 of SemEval 2022: Multilingual Idiomaticity Detection and Sentence Embedding (Tayyar Madabushi et al, 2022).…”

“…In the test set, there are 916 English, 713 Portuguese, and 713 Galician examples, and macro F1 score is used as an evaluation metric. It should be noted that the dataset provided by TayyarMadabushi et al (2022) consists of four data splits: The training set, two development sets and the test set. Of the two development sets, the first -called the 'dev' split -includes gold labels and the secondcalled the 'eval' split -does not include gold labels but requires submission to the competition website.…”

SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding

“…We use the training and development splits from Tayyar Madabushi et al (2021) with the addition of Galician data, and use the test split released by them as the evaluation split during the initial practice phase of the competition. We create an independent test set consisting of examples with new MWEs, and this set was used to determine the teams' final rankings.…”

HIT at SemEval-2022 Task 2: Pre-trained Language Model for Idioms Detection

“…Understanding the semantic meaning of a sentence requires the correct identification of the MWE in the sentence. Table 1 contains one case for each of the two usages (Tayyar Madabushi et al, 2021).…”