The Winograd Schema Challenge, the task of resolving pronouns in certain carefully-structured sentences, has received considerable interest in the past few years as an alternative to the Turing Test. Systems developed to tackle this challenge have typically been evaluated on a small set of hand-crafted collections of sentences, since the development of new sentences by individuals is itself a rather challenging task, requiring care and creativity. In this paper we approach the problem of developing Winograd schemas via the introduction of WinoFlexi, a flexible online crowdsourcing system. Our empirical evaluation of the system's performance suggests that WinoFlexi allows crowdworkers to develop Winograd schemas of quality similar to that of most typical existing collections.

The Winograd Schema Challenge-the task of resolving pronouns in certain carefully-constructed sentences-has recently been proposed as a basis for a novel form of CAPTCHAs. Such uses of the task necessitate the availability of a large, and presumably continuously-replenished, collection of available Winograd Schemas, which goes beyond what human experts can reasonably develop by themselves. Towards tackling this issue, we introduce Winventor, the first, to our knowledge, system that attempts to fully automate the development of Winograd Schemas, or at least to considerably help humans in this development task. Beyond describing the system, the paper presents a series of three studies that demonstrate, respectively, Winventor's ability to replicate existing Winograd Schemas from the literature, automatically develop reasonable Winograd Schemas from scratch, and aid humans in developing Winograd Schemas by post-processing the system's suggestions.

CAPTCHAs have established themselves as a standard technology to confidently distinguish humans from bots. Beyond the typical use for security reasons, CAPTCHAs have helped promote AI research in challenge tasks such as image classification and optical character recognition. It is, therefore, natural to consider what other challenge tasks for AI could serve a role in CAPTCHAs. The Winograd Schema Challenge (WSC), a certain form of hard pronoun resolution tasks, was proposed by Levesque as such a challenge task to promote research in AI. Based on current reports in the literature, the WSC remains a challenging task for bots, and is, therefore, a candidate to serve as a form of CAPTCHA. In this work we investigate whether this a priori appropriateness of the WSC as a form of CAPTCHA can be justified in terms of its acceptability by the human users in relation to existing CAPTCHA tasks. Our empirical study involved a total of 329 students, aged between 11 and 15, and showed that the WSC is generally faster and easier to solve than, and equally entertaining with, the most typical existing CAPTCHA tasks.

The Winograd Schema Challenge (WSC) — the task of resolving pronouns in certain sentences where shallow parsing techniques seem not to be directly applicable — has been proposed as an alternative to the Turing Test. According to Levesque, having access to a large corpus of text would likely not help much in the WSC. Among a number of attempts to tackle this challenge, one particular approach has demonstrated the plausibility of using commonsense knowledge automatically acquired from raw text in English Wikipedia.Here, we present the results of a large-scale experiment that shows how the performance of that particular automated approach varies with the availability of training material. We compare the results of this experiment with two studies: one from the literature that investigates how adult native speakers tackle the WSC, and one that we design and undertake to investigate how teenager non-native speakers tackle the WSC. We find that the performance of the automated approach correlates positively with the performance of humans, suggesting that the performance of the particular automated approach could be used as a metric of hardness for WSC instances.

No abstract