Textual entailment as an evaluation framework for metaphor resolution

Agerri, Rodrigo; Barnden, John A.; Lee, Mark; Wallington, Alan M.

doi:10.3115/1626481.1626509

Cited by 2 publications

(1 citation statement)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Given that figurativeness is commonplace in everyday communication (Lakoff and Johnson, 2008), progress in the field of Natural Language Understanding (NLU) would be incomplete without figurativeness understanding. Consequently, figurative text has been studied in various downstream NLP tasks such as machine translation (Dankers et al, 2022), textual entailment (Agerri, 2008), (Chakrabarty et al, 2021), (Liu et al, 2022) and dialog models (Jhamtani et al, 2021), inter-alia. However, to the best of our knowledge, there has not been a systematic study of figurative language understanding capabilities of question answering models.…”

Section: -Oscar Wildementioning

confidence: 99%

Does the “most sinfully decadent cake ever” taste good? Answering Yes/No Questions from Figurative Contexts

Rakshit,

Flanigan

2023

Proceedings of the Conference Recent Advances in Natural Language Processing - Large Language Models for Natural Language Proce

View full text Add to dashboard Cite

Figurative language is commonplace in natural language, and while making communication memorable and creative, can be difficult to understand. In this work, we investigate the robustness of Question Answering (QA) models on figurative text. Yes/no questions, in particular, are a useful probe of figurative language understanding capabilities of large language models. We propose FigurativeQA, a set of 1000 yes/no questions with figurative and nonfigurative contexts, extracted from the domains of restaurant and product reviews. We show that state-of-the-art BERT-based QA models exhibit an average performance drop of up to 15% points when answering questions from figurative contexts, as compared to non-figurative ones. While models like GPT-3 and ChatGPT are better at handling figurative texts, we show that further performance gains can be achieved by automatically simplifying the figurative contexts into their non-figurative (literal) counterparts. We find that the best overall model is ChatGPT with chain-of-thought prompting to generate non-figurative contexts. Our work provides a promising direction for building more robust QA models with figurative language understanding capabilities.

show abstract

Section: -Oscar Wildementioning

confidence: 99%

Does the “most sinfully decadent cake ever” taste good? Answering Yes/No Questions from Figurative Contexts

Rakshit,

Flanigan

2023

Proceedings of the Conference Recent Advances in Natural Language Processing - Large Language Models for Natural Language Proce

View full text Add to dashboard Cite

show abstract

Figurative Language in Recognizing Textual Entailment

Chakrabarty¹,

Ghosh²,

Poliak³

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

We introduce a collection of recognizing textual entailment (RTE) datasets focused on figurative language. We leverage five existing datasets annotated for a variety of figurative language -simile, metaphor, and irony -and frame them into over 12,500 RTE examples.We evaluate how well state-of-the-art models trained on popular RTE datasets capture different aspects of figurative language. Our results and analyses indicate that these models might not sufficiently capture figurative language, struggling to perform pragmatic inference and reasoning about world knowledge. Ultimately, our datasets provide a challenging testbed for evaluating RTE models.

show abstract

Textual entailment as an evaluation framework for metaphor resolution

Cited by 2 publications

References 13 publications

Does the “most sinfully decadent cake ever” taste good? Answering Yes/No Questions from Figurative Contexts

Does the “most sinfully decadent cake ever” taste good? Answering Yes/No Questions from Figurative Contexts

Figurative Language in Recognizing Textual Entailment

Contact Info

Product

Resources

About