Semantic noise in the Winograd Schema Challenge of pronoun disambiguation

Abstract

The Winograd Schema Challenge (WSC) of pronoun disambiguation is a Natural Language Processing (NLP) task designed to test to what extent the reading comprehension capabilities of language models (LMs) can be compared to those of human subjects. It is generally assumed across the NLP literature that human subjects are capable of resolving this task because of their acquired commonsense knowledge, thus setting a commonsense benchmark for LMs, one which has even been proposed as an alternative to the Turing test. In the context of complex natural language communications, Shannon and Weaver observed that the act of semantic interpretation is subject to semantic noise (Shannon and Weaver, 1964 (1949)). Semantic noise is a constraint that ensues from terms exhibiting variable interpretations across contexts, presenting a challenge to the resolution of tasks such as the WSC. However, the main argument of this paper is that rather than seeing semantic noise as a challenge to otherwise unambiguous communication, it can also be understood as a functional quality of natural language, given that it results in the conceptual negotiation of terms. Failing to theoretically attend to this linguistic matter of fact leads to unintended problems in instances where NLP applications are offered as unbiased or objectively applicable solutions. To address this, this article offers a renewed and original analysis of a series of Winograd Schemas, in order to demonstrate how they are not as straightforwardly solvable by human subjects as is commonly claimed across the NLP literature. The methodology employed is that of historical contextualisation in information theory, and qualitative cultural analysis drawing on examples from a wide variety of recent NLP literature.