Pay Attention to the Ending:Strong Neural Baselines for the ROC
            Story Cloze Task

Cai, Zheng; Tu, Lifu; Gimpel, Kevin

doi:10.18653/v1/p17-2097

Cited by 56 publications

(65 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In ROC Stories (Mostafazadeh et al, 2016), a story cloze dataset, Schwartz et al (2017b) obtained a high performance by only considering the candidate endings, without even looking at the story context. In this case, stylistic features of the candidate endings alone, such as the length or certain words, were strong indicators of the correct ending (Schwartz et al, 2017a;Cai et al, 2017). A similar phenomenon was observed in reading comprehension, where systems performed non-trivially well by using only the final sentence in the passage or ignoring the passage altogether (Kaushik & Lipton, 2018).…”

Section: Fine-tuning On Target Datasetsmentioning

confidence: 52%

Don’t Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference

Belinkov¹,

Poliak²,

Shieber³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

Natural Language Inference (NLI) datasets often contain hypothesis-only biases-artifacts that allow models to achieve non-trivial performance without learning whether a premise entails a hypothesis. We propose two probabilistic methods to build models that are more robust to such biases and better transfer across datasets. In contrast to standard approaches to NLI, our methods predict the probability of a premise given a hypothesis and NLI label, discouraging models from ignoring the premise. We evaluate our methods on synthetic and existing NLI datasets by training on datasets containing biases and testing on datasets containing no (or different) hypothesis-only biases. Our results indicate that these methods can make NLI models more robust to dataset-specific artifacts, transferring better than a baseline architecture in 9 out of 12 NLI datasets. Additionally, we provide an extensive analysis of the interplay of our methods with known biases in NLI datasets, as well as the effects of encouraging models to ignore biases and fine-tuning on target datasets. 1 * * Equal contribution 1 Our code is available at https://github.com/ azpoliak/robust-nli.2 This hypothesis contradicts the premise and would likely not be inferred.

show abstract

Section: Fine-tuning On Target Datasetsmentioning

confidence: 52%

Don’t Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference

Belinkov¹,

Poliak²,

Shieber³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…A: I was getting tired of having food in the house. tion of the problem (Gururangan et al, 2018;Cai et al, 2017). Therefore, we report problem ablation study in Table 4 using BERT-FT as a simple but powerful straw man approach.…”

Section: Results and Analysismentioning

confidence: 99%

Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning

Huang¹,

Bras²,

Bhagavatula³

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

229

179

View full text Add to dashboard Cite

Understanding narratives requires reading between the lines, which in turn, requires interpreting the likely causes and effects of events, even when they are not mentioned explicitly. In this paper, we introduce COSMOS QA, a large-scale dataset of 35, 600 problems that require commonsense-based reading comprehension, formulated as multiple-choice questions. In stark contrast to most existing reading comprehension datasets where the questions focus on factual and literal understanding of the context paragraph, our dataset focuses on reading between the lines over a diverse collection of people's everyday narratives, asking such questions as "what might be the possible reason of ...?", or "what would have happened if ..." that require reasoning beyond the exact text spans in the context. To establish baseline performances on COSMOS QA, we experiment with several state-of-the-art neural architectures for reading comprehension, and also propose a new architecture that improves over the competitive baselines. Experimental results demonstrate a significant gap between machine (68.4%) and human performance (94%), pointing to avenues for future research on commonsense machine comprehension. Dataset, code and leaderboard is publicly available at https://wilburone.

show abstract

“…To improve the performance, fea-tures like topic words and sentiment score are also extracted and incorporated (Chaturvedi, Peng, and Roth 2017). Neural network models have also been applied to this task (e.g., (Huang et al 2013) and (Cai, Tu, and Gimpel 2017)), which use LSTM to encode different parts of the story and calculate their similarities. In addition, (Li et al 2018) introduces event frame to their model and leverages five different embeddings.…”

Section: Related Workmentioning

confidence: 99%

“…We use the following models as our baselines: 2 http://cs.rochester.edu/nlp/rocstories Model Accuracy(%) Msap (Schwartz et al 2017) 75.2 HCM (Chaturvedi, Peng, and Roth 2017) 77.6 DSSM (Huang et al 2013) 58.5 Cai (Cai, Tu, and Gimpel 2017) 74.7 SeqMANN (Li et al 2018) 84.7 FTLM (Radford et al 2018 86.5 Our Model(Plot&End) 78.4 Our Model(Full Story) 87.6* Table 2: Performance comparison with baselines, *indicates that the model is significantly better than best baseline model…”

Section: Experiments Baselinesmentioning

confidence: 99%

See 1 more Smart Citation

Incorporating Structured Commonsense Knowledge in Story Completion

Chen

Zhou

2019

AAAI

View full text Add to dashboard Cite

The ability to select an appropriate story ending is the first step towards perfect narrative comprehension. Story ending prediction requires not only the explicit clues within the context, but also the implicit knowledge (such as commonsense) to construct a reasonable and consistent story. However, most previous approaches do not explicitly use background commonsense knowledge. We present a neural story ending selection model that integrates three types of information: narrative sequence, sentiment evolution and commonsense knowledge. Experiments show that our model outperforms state-ofthe-art approaches on a public dataset, ROCStory Cloze Task (Mostafazadeh et al. 2017), and the performance gain from adding the additional commonsense knowledge is significant.

show abstract

Pay Attention to the Ending:Strong Neural Baselines for the ROC Story Cloze Task

Cited by 56 publications

References 17 publications

Don’t Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference

Don’t Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference

Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning

Incorporating Structured Commonsense Knowledge in Story Completion

Contact Info

Product

Resources

About