Social IQa: Commonsense Reasoning about Social Interactions

Abstract

We introduce SOCIAL IQA, the first largescale benchmark for commonsense reasoning about social situations. SOCIAL IQA contains 38,000 multiple choice questions for probing emotional and social intelligence in a variety of everyday situations (e.g., Q: "Jordan wanted to tell Tracy a secret, so Jordan leaned towards Tracy. Why did Jordan do this?" A: "Make sure no one else could hear"). Through crowdsourcing, we collect commonsense questions along with correct and incorrect answers about social interactions, using a new framework that mitigates stylistic artifacts in incorrect answers by asking workers to provide the right answer to a different but related question. Empirical results show that our benchmark is challenging for existing question-answering models based on pretrained language models, compared to human performance (>20% gap). Notably, we further establish SOCIAL IQA as a resource for transfer learning of commonsense knowledge, achieving state-of-the-art performance on multiple commonsense reasoning tasks (Winograd Schemas, COPA).

Social Bias Frames: Reasoning about Social and Power Implications of Language

“…The model generates the correct stereotypes when there is high lexical overlap with the post (e.g., examples d and e). This is in line with previous research showing that large language models rely on correlational patterns in data (Sap et al, 2019c;Sakaguchi et al, 2020).…”

How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG

“…Based on the protocols we introduce, we show that performing at state-of-theart on these datasets does not necessarily imply strong common-sense reasoning capability. We are happy to see a rising interest in the WSC in the community, including very recent work by Ruan et al (2019) and Sap et al (2019), which reinforces the need for proper evaluation protocols. With the release of an increasing number of finegrained inference tasks aimed at these abilities (Roemmele et al, 2011;Morgenstern et al, 2016;Wang et al, 2018;Rashkin et al, 2018;McCann et al, 2018), the issue of experimental validity in CSR will also become even more important.…”

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts

“…But despite these impressive performance improvements in a variety of NLP tasks, it remains unclear whether these models are performing complex reasoning, or if they are merely learning complex surface correlation patterns (Davis and Marcus, 2015;Marcus, 2018). This difficulty in measuring the progress in commonsense reasoning using downstream tasks has yielded increased efforts at developing robust benchmarks for directly measuring commonsense capabilities in multiple settings, such as social interactions (Sap et al, 2019b;Rashkin et al, 2018a) and physical situations (Zellers et al, 2019;Talmor et al, 2019).…”

“…In response, recent work has focused on using crowdsourcing and automatic filtering to design large-scale benchmarks while maintaining negative examples that are adversarial to machines (Zellers et al, 2018). We will review recent benchmarks that have emerged to assess whether machines have acquired physical (e.g., Talmor et al, 2019;Zellers et al, 2019), social (e.g., Sap et al, 2019b), or temporal commonsense reasoning capabilities (e.g., , as well as benchmarks that combine commonsense abilities with other tasks (e.g., reading comprehension; Ostermann et al, 2018;.…”

“…Due to the research interests and output of the presenters, we estimate that approximately 30% of the tutorial will center around work done by the presenters (Rashkin et al, 2018b;Sap et al, 2019a;Rashkin et al, 2018a;Sap et al, 2019b;Zellers et al, 2018Zellers et al, , 2019Sakaguchi et al, 2019;Shwartz et al, 2020).…”