A Knowledge Hunting Framework for Common Sense Reasoning

Emami, Ali; Cruz, Noelia De La; Trischler, Adam; Suleman, Kaheer; Cheung, Jackie Kit

doi:10.48550/arxiv.1810.01375

Cited by 3 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A plethora of works in the literature focused on the development of systems for tackling the WSC. In this regard, Emami et al [15] developed a rule-based system that, by focusing on knowledge-hunting on the Web, was able to achieve better than 57% accuracy on the original WSC problem (WSC273). According to Kocijan et al [32], this was the first approach to achieve better than chance accuracy.…”

Section: Related Workmentioning

confidence: 99%

Experience and prediction: a metric of hardness for a novel litmus test

Isaak

Michael

2021

Journal of Logic and Computation

View full text Add to dashboard Cite

In the past decade, the Winograd schema challenge (WSC) has become a central aspect of the research community as a novel litmus test. Consequently, the WSC has spurred research interest because it can be seen as the means to understand human behavior. In this regard, the development of new techniques has made possible the usage of Winograd schemas in various fields, such as the design of novel forms of CAPTCHAs. Work from the literature that established a baseline for human adult performance on the WSC has shown that not all schemas are the same, meaning that they could potentially be categorized according to their perceived hardness for humans. In this regard, this hardness metric could be used in future challenges or in the WSC CAPTCHA service to differentiate between Winograd schemas. Recent work of ours has shown that this could be achieved via the design of an automated system that is able to output the hardness indexes of Winograd schemas, albeit with limitations regarding the number of schemas it could be applied on. This paper adds to previous research by presenting a new system that is based on machine learning, able to output the hardness of any Winograd schema faster and more accurately than any other previously used method. Our developed system, which works within two different approaches, namely the random forest and deep learning (LSTM-based), is ready to be used as an extension of any other system that aims to differentiate between Winograd schemas, according to their perceived hardness for humans. At the same time, along with our developed system we extend previous work by presenting the results of a large-scale experiment that shows how human performance varies across Winograd schemas.

show abstract

Section: Related Workmentioning

confidence: 99%

Experience and prediction: a metric of hardness for a novel litmus test

Isaak

Michael

2021

Journal of Logic and Computation

View full text Add to dashboard Cite

show abstract

“…Note that the results for (Ruan et al 2019) are fine-tuned on the whole WSCR dataset, including the training and test sets. Results for LM ensemble (Trinh and Le 2018) and Knowl-edge Hunter (Emami et al 2018) are taken from (Trichelair et al 2018). Results for "BERT large + MTP" is taken from (Kocijan et al 2019) as the baseline of applying BERT to the WSC task.…”

Section: Winograd Schema Challengementioning

confidence: 99%

Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models

Ye,

Chen,

Wang

et al. 2019

Preprint

View full text Add to dashboard Cite

Neural language representation models such as Bidirectional Encoder Representations from Transformers (BERT) pretrained on large-scale corpora can well capture rich semantics from plain text, and can be fine-tuned to consistently improve the performance on various natural language processing (NLP) tasks. However, the existing pre-trained language representation models rarely consider explicitly incorporating commonsense knowledge or other knowledge. In this paper, we develop a pre-training approach for incorporating commonsense knowledge into language representation models. We construct a commonsense-related multi-choice question answering dataset for pre-training a neural language representation model. The dataset is created automatically by our proposed "align, mask, and select" (AMS) method. We also investigate different pre-training tasks. Experimental results demonstrate that pre-training models using the proposed approach followed by fine-tuning achieves significant improvements on various commonsense-related tasks, such as Com-monsenseQA and Winograd Schema Challenge, while maintaining comparable performance on other NLP tasks, such as sentence classification and natural language inference (NLI) tasks, compared to the original BERT models.

show abstract

“…Therefore, most of the pronoun resolution models that address hard pronoun resolution rely on little (Liu et al, 2019) or no training data, via unsupervised pre-training (Trinh and Le, 2018;Radford et al, 2019). Another approach involves using external knowledge bases (Emami et al, 2018;Fähndrich et al, 2018), however, the accuracy of these models still lags behind that of the aforementioned pre-trained models.…”

Section: Related Workmentioning

confidence: 99%

WikiCREM: A Large Unsupervised Corpus for Coreference Resolution

Kocijan¹,

Camburu²,

Creţu³

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

Pronoun resolution is a major area of natural language understanding. However, large-scale training sets are still scarce, since manually labelling data is costly. In this work, we introduce WIKICREM (Wikipedia CoREferences Masked) a large-scale, yet accurate dataset of pronoun disambiguation instances. We use a language-model-based approach for pronoun resolution in combination with our WI-KICREM dataset. We compare a series of models on a collection of diverse and challenging coreference resolution problems, where we match or outperform previous state-of-theart approaches on 6 out of 7 datasets, such as GAP, DPR, WNLI, PDP, WINOBIAS, and WINOGENDER. We release our model to be used off-the-shelf for solving pronoun disambiguation. 1 The code can be found at https://github.com/ vid-koci/bert-commonsense. The dataset and the models can be obtained from

show abstract

A Knowledge Hunting Framework for Common Sense Reasoning

Cited by 3 publications

References 0 publications

Experience and prediction: a metric of hardness for a novel litmus test

Experience and prediction: a metric of hardness for a novel litmus test

Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models

WikiCREM: A Large Unsupervised Corpus for Coreference Resolution

Contact Info

Product

Resources

About