WinoLogic: A Zero-Shot Logic-based Diagnostic Dataset for Winograd Schema Challenge

He, Weinan; Huang, Canming; Liu, Yongmei; Zhu, Xiaodan

doi:10.18653/v1/2021.emnlp-main.307

Cited by 10 publications

(5 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…sentences Winograd Schema Pronoun resolution 285 sentences Expert construction Challenge (WSC) [82] Winogrande [115] Cloze task 44,000 Crowd sourcing. Winologic [58] Logical explanation 273 problems Expert construction of pronoun resolution Winowhy [157] Justify pronoun 276 questions Crowd sourcing resolution Winventor [66] Pronoun resolution 848 sentences Synthesized. WordCraft [68] Construct new 3147 combinations Synthesized from entity from old ones existing resource ZUCC [153] Predict next observation 14,000 examples Synthesized in interactive fiction from interactive fiction…”

Section: Taskmentioning

confidence: 99%

Benchmarks for Automated Commonsense Reasoning: A Survey

Davis¹

2023

Preprint

View full text Add to dashboard Cite

More than one hundred benchmarks have been developed to test the commonsense knowledge and commonsense reasoning abilities of artificial intelligence (AI) systems. However, these benchmarks are often flawed and many aspects of common sense remain untested. Consequently, we do not currently have any reliable way of measuring to what extent existing AI systems have achieved these abilities.This paper surveys the development and uses of AI commonsense benchmarks. We discuss the nature of common sense; the role of common sense in AI; the goals served by constructing commonsense benchmarks; and desirable features of commonsense benchmarks. We analyze the common flaws in benchmarks, and we argue that it is worthwhile to invest the work needed ensure that benchmark examples are consistently high quality. We survey the various methods of constructing commonsense benchmarks. We enumerate 139 commonsense benchmarks that have been developed: 102 text-based, 18 image-based, 12 video based, and 7 simulated physical environments. We discuss the gaps in the existing benchmarks and aspects of commonsense reasoning that are not addressed in any existing benchmark. We conclude with a number of recommendations for future development of commonsense AI benchmarks.

show abstract

Section: Taskmentioning

confidence: 99%

Benchmarks for Automated Commonsense Reasoning: A Survey

Davis¹

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Algumas alternativas encontradas para a ampliação do volume de esquemas são o Winoflexi [Isaak;Michael, 2019], que utiliza crowdsourcing para o desenvolvimento de novas sentenças e o Winventor [Nicos;Michael, 2020] que busca automatizar a criação de esquemas. No Winologic [He et al, 2021] novas frases foram construídas utilizando teoremas lógicos.…”

Section: Winograd E a Evolução Dos Benchmarksunclassified

Avaliação do senso comum em modelos de linguagem através de benchmarks: Desafio de Winograd aplicado ao ChatGPT em português brasileiro

do Nascimento,

Cortiz

2023

Anais Do XIV Simpósio Brasileiro De Tecnologia Da Informação E Da Linguagem Humana (STIL 2023)

View full text Add to dashboard Cite

O desempenho em benchmarks é apresentado como uma forma de avaliação efetiva dos limites de compreensão dos modelos de linguagem. Neste sentido, o desafio de esquemas de Winograd, que se propõe a avaliar o senso comum por meio de tarefas de desambiguação de pronomes, deu origem a diferentes métricas e datasets. Ao aplicar a tradução do desafio de Winograd ao ChatGPT em português brasileiro, identificamos resultados equiparáveis aos obtidos em inglês. Contudo, é preciso ter cautela ao interpretar estes dados, visto que existem vieses associados ao treinamento dos modelos e lacunas quanto às dimensões de raciocínio contempladas pelos métodos de avaliação disponíveis.

show abstract

“…WinoWhy and WinoLogic (He et al, 2021) are collections of correct and incorrect natural language explanations of all Winograd Schemas in the Wsc273 dataset. The goal of both datasets is to determine whether systems that can correctly answer Winograd Schemas are also capable to identify the correct explanations for their choice.…”

Section: Datasets Of Explanationsmentioning

confidence: 99%

The Defeat of the Winograd Schema Challenge

Kocijan¹,

Davis²,

Lukasiewicz³

et al. 2022

Preprint

View full text Add to dashboard Cite

The Winograd Schema Challenge-a set of twin sentences involving pronoun reference disambiguation that seem to require the use of commonsense knowledge-was proposed by Hector Levesque in 2011. By 2019, a number of AI systems, based on large pre-trained transformer-based language models and fine-tuned on these kinds of problems, achieved better than 90% accuracy. In this paper, we review the history of the Winograd Schema Challenge and assess its significance.

show abstract

WinoLogic: A Zero-Shot Logic-based Diagnostic Dataset for Winograd Schema Challenge

Cited by 10 publications

References 19 publications

Benchmarks for Automated Commonsense Reasoning: A Survey

Benchmarks for Automated Commonsense Reasoning: A Survey

Avaliação do senso comum em modelos de linguagem através de benchmarks: Desafio de Winograd aplicado ao ChatGPT em português brasileiro

The Defeat of the Winograd Schema Challenge

Contact Info

Product

Resources

About