Can a Gorilla Ride a Camel? Learning Semantic Plausibility from Text

Porada, Ian; Suleman, Kaheer; Cheung, Jackie Chi Kit

doi:10.18653/v1/d19-6015

Cited by 10 publications

(12 citation statements)

References 36 publications

(51 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other studies introduced larger datasets, but focused on more specific notions of event plausibility (e.g. the plausibility depending on the physical properties of the participants) (Wang et al, 2018;Porada et al, 2019;Ko et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge

Pedinotti¹,

Rambelli²,

Chersoni³

et al. 2021

Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics

View full text Add to dashboard Cite

Prior research has explored the ability of computational models to predict a word semantic fit with a given predicate. While much work has been devoted to modeling the typicality relation between verbs and arguments in isolation, in this paper we take a broader perspective by assessing whether and to what extent computational approaches have access to the information about the typicality of entire events and situations described in language (Generalized Event Knowledge).Given the recent success of Transformers Language Models (TLMs), we decided to test them on a benchmark for the dynamic estimation of thematic fit. The evaluation of these models was performed in comparison with SDM, a framework specifically designed to integrate events in sentence meaning representations, and we conducted a detailed error analysis to investigate which factors affect their behavior. Our results show that TLMs can reach performances that are comparable to those achieved by SDM. However, additional analysis consistently suggests that TLMs do not capture important aspects of event knowledge, and their predictions often depend on surface linguistic features, such as frequent words, collocations and syntactic patterns, thereby showing sub-optimal generalization abilities.

show abstract

Section: Related Workmentioning

confidence: 99%

Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge

Pedinotti¹,

Rambelli²,

Chersoni³

et al. 2021

Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics

View full text Add to dashboard Cite

show abstract

“…We use English Wikipedia to construct the selfsupervised training data. As a relatively clean, definitional corpus, plausibility models trained on Wikipedia have been shown to correlate with human judgements better than those trained on similarly sized corpora (Zhang et al, 2019a;Porada et al, 2019).…”

Section: Training Datamentioning

confidence: 99%

“…Annotators were instructed to ignore possible metaphorical meanings of an event. We divide the dataset equally into a validation and test set following the split of Porada et al (2019).…”

Section: Pep-3kmentioning

confidence: 99%

Modeling Event Plausibility with Consistent Conceptual Abstraction

Porada¹,

Suleman

Trischler

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

Understanding natural language requires common sense, one aspect of which is the ability to discern the plausibility of events. While distributional models-most recently pre-trained, Transformer language modelshave demonstrated improvements in modeling event plausibility, their performance still falls short of humans'. In this work, we show that Transformer-based plausibility models are markedly inconsistent across the conceptual classes of a lexical hierarchy, inferring that "a person breathing" is plausible while "a dentist breathing" is not, for example. We find this inconsistency persists even when models are softly injected with lexical knowledge, and we present a simple post-hoc method of forcing model consistency that improves correlation with human plausibility judgements.

show abstract

“…Some datasets focus on non-sentential eventual plausibility (Wang et al, 2018;Porada et al, 2019), such as "gorilla-ride-camel". In contrast, our dataset is based on statements which includes events, descriptions, assertion etc, not merely events, such as "China's territory is larger than Japan's".…”

Section: Related Workmentioning

confidence: 99%

SemEval-2020 Task 4: Commonsense Validation and Explanation

Wang¹,

Liang²,

Jin³

et al. 2020

Proceedings of the Fourteenth Workshop on Semantic Evaluation

View full text Add to dashboard Cite

In this paper, we present SemEval-2020 Task 4, Commonsense Validation and Explanation (ComVE), which includes three subtasks, aiming to evaluate whether a system can distinguish a natural language statement that makes sense to humans from one that does not, and provide the reasons. Specifically, in our first subtask, the participating systems are required to choose from two natural language statements of similar wording the one that makes sense and the one does not. The second subtask additionally asks a system to select the key reason from three options why a given statement does not make sense. In the third subtask, a participating system needs to generate the reason. We finally attracted 39 teams participating at least one of the three subtasks. For Subtask A and Subtask B, the performances of top-ranked systems are close to that of humans. However, for Subtask C, there is still a relatively large gap between systems and human performance. The dataset used in our task can be found at https://github.com/wangcunxiang/SemEval2020-Task4-Commonsense-Validation-and-Explanation; The leaderboard can be found at https://competitions.codalab.org/competitions/21080#results.

show abstract

Can a Gorilla Ride a Camel? Learning Semantic Plausibility from Text

Cited by 10 publications

References 36 publications

Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge

Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge

Modeling Event Plausibility with Consistent Conceptual Abstraction

SemEval-2020 Task 4: Commonsense Validation and Explanation

Contact Info

Product

Resources

About