Commonsense Interpretation of Triangle Behavior

Gordon, Andrew S.

doi:10.1609/aaai.v30i1.9881

Cited by 21 publications

(3 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…in interactive text simulated world Social IQA [117] Questions testing 37,000 questions Crowd sourcing social intelligence Spatial Spatial reasoning 1448 questions Expert construction Commonsense [92] yes/no questions SWAG [154] What happens next? 113,000 questions Crowd sourcing TG-CSR [116] Question answering 331 questions Expert construction TimeDial [109] Cloze tasks for 1,100 dialogues Crowd sourcing temporal reasoning Torque [103] Order of events 3,200 news stories Crowd sourcing in a news story 21,000 questions Tracie [161] Order of implicit 5500 problems Crowd sourcing and explicit event Triangle COPA [51] Why did that thing 100 examples Expert construction do that? TRIP [129] Which story is more 2100 stories Crowd sourcing plausible?…”

Section: Taskmentioning

confidence: 99%

Benchmarks for Automated Commonsense Reasoning: A Survey

Davis¹

2023

Preprint

View full text Add to dashboard Cite

More than one hundred benchmarks have been developed to test the commonsense knowledge and commonsense reasoning abilities of artificial intelligence (AI) systems. However, these benchmarks are often flawed and many aspects of common sense remain untested. Consequently, we do not currently have any reliable way of measuring to what extent existing AI systems have achieved these abilities.This paper surveys the development and uses of AI commonsense benchmarks. We discuss the nature of common sense; the role of common sense in AI; the goals served by constructing commonsense benchmarks; and desirable features of commonsense benchmarks. We analyze the common flaws in benchmarks, and we argue that it is worthwhile to invest the work needed ensure that benchmark examples are consistently high quality. We survey the various methods of constructing commonsense benchmarks. We enumerate 139 commonsense benchmarks that have been developed: 102 text-based, 18 image-based, 12 video based, and 7 simulated physical environments. We discuss the gaps in the existing benchmarks and aspects of commonsense reasoning that are not addressed in any existing benchmark. We conclude with a number of recommendations for future development of commonsense AI benchmarks.

show abstract

Section: Taskmentioning

confidence: 99%

Benchmarks for Automated Commonsense Reasoning: A Survey

Davis¹

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…This design eliminates the vision challenges associated with naturalistic scenes, probing a machine's ability to learn higher-level cognitive representations. (Gordon, 2016;Springer, Meier, & Berry, 1996). Moreover, our design streamlines the engineering process for synthesizing the thousands of videos necessary to train and test models.…”

Section: Benchmark Tasksmentioning

confidence: 99%

An Infant-Cognition Inspired Machine Benchmark for Identifying Agency, Affiliation, Belief, and Intention

Li,

Yasuda,

Dillon

et al. 2023

Preprint

View full text Add to dashboard Cite

Preverbal infants have remarkable abilities to understand others' intentions, beliefs, and social affiliations. These skills lay the groundwork for complex cognitive and language development, crucial for navigating human social dynamics throughout life. In contrast, recent developments in Artificial Intelligence systems (AI) are often designed to emulate human-like behaviors by extracting common sense knowledge from language or sensory data. Despite their impressive performance in many aspects, they fail to recover some foundational theory of mind capacities found in early infancy. This discrepancy highlights the critical gap in creating AI that understands people and thinks like people. To address this, we introduce a suite of theory of mind tasks to directly compare the capacities of infants and AI to understand others. Expanding on our prior work, Baby Intuition Benchmark (BIB), which challenges machines to understand goal-directed behaviors, this work expands to a broader scope of social cognition, probing machine understandings of social affiliations, goal attribution, and false belief. We evaluate BIB and the new tasks using Transformer models trained with a self-supervised learning paradigm. The model shows improved performance over existing baselines, elevating the lower bound of social reasoning capacities in learning-driven machines. However, it still exposes the limitations of learning complex causal relations and complex human mental states through behavioral data alone, underscoring the challenges in achieving a human-like theory of mind reasoning in AI.

show abstract

“…Intentions have been extensively explored in psychology tests, e.g., behavioral re-enactment (Meltzoff, 1995), action prediction (Phillips et al, 2002, intention explanation (Smiley, 2001), and intention attribution to abstract figures (Castelli, 2006). EPISTEMIC REASONING (Cohen, 2021) Infer T -✓ ✓ TOMI (Nematzadeh et al, 2018) QA T -✓ ✓ ✓ HI-TOM (He et al, 2023) QA T -✓ ✓ ✓ MINDGAMES (Sileo and Lernould, 2023) Infer T -✓ ✓ ✓ ✓ ADV-CSFB (Shapira et al, 2023a) QA H -✓ ✓ CONVENTAIL (Zhang and Chai, 2010) Infer H -✓ ✓ ✓ ✓ SOCIALIQA (Sap et al, 2019) QA H -✓ ✓ ✓ BEST (Tracey et al, 2022) -H -✓ ✓ ✓ ✓ FAUXPAS-EAI (Shapira et al, 2023b) QA H,AI -✓ ✓ ✓ COKE NLG AI -✓ ✓ ✓ ✓ TOM-IN-AMC Infer H -✓ ✓ ✓ ✓ G4C (Zhou et al, 2023b) NLG H,AI -✓ ✓ ✓ ✓ ✓ ✓ VISUALBELIEFS (Eysenbach et al, 2016) Infer -Cartoon ✓ ✓ ✓ TRIANGLE COPA (Gordon, 2016) QA H Cartoon ✓ ✓ ✓ ✓ MSED (Jia et al, 2022) Infer H Images ✓ ✓ ✓ BIB (Gandhi et al, 2021) Infer -2D Grid ✓ ✓ ✓ AGENT (Shu et al, 2021) Infer -3D Sim. ✓ ✓ ✓ ✓ MTOM (Rabinowitz et al, 2018) Infer -2D Grid ✓ ✓ ✓ SYMMTOM (Sclar et al, 2022) MARL -2D Grid ✓ ✓ ✓ ✓ ✓ ✓ MINDCRAFT (Bara et al, 2021) Infer (Bara et al, 2023) Infer Desires.…”

Section: Abilities In Theory Of Mind Space (Atoms) Frameworkmentioning

confidence: 99%

Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models

Ma,

Sansom,

Peng

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Large Language Models (LLMs) have generated considerable interest and debate regarding their potential emergence of Theory of Mind (ToM). Several recent inquiries reveal a lack of robust ToM in these models and pose a pressing demand to develop new benchmarks, as current ones primarily focus on different aspects of ToM and are prone to shortcuts and data leakage. In this position paper, we seek to answer two road-blocking questions: (1) How can we taxonomize a holistic landscape of machine ToM? (2) What is a more effective evaluation protocol for machine ToM? Following psychological studies, we taxonomize machine ToM into 7 mental state categories and delineate existing benchmarks to identify under-explored aspects of ToM. We argue for a holistic and situated evaluation of ToM to break ToM into individual components and treat LLMs as an agent who is physically situated in environments and socially situated in interactions with humans. Such situated evaluation provides a more comprehensive assessment of mental states and potentially mitigates the risk of shortcuts and data leakage. We further present a pilot study in a grid world setup as a proof of concept. We hope this position paper can facilitate future research to integrate ToM with LLMs and offer an intuitive means for researchers to better position their work in the landscape of ToM. of faux pas by normally developing children and children with asperger syndrome or high-functioning autism. Journal of autism and developmental disorders, 29:407-418.Cindy Beaudoin, Élizabel Leblanc, Charlotte Gagner, and Miriam H Beauchamp. 2020. Systematic review and inventory of theory of mind measures for young children. Frontiers in psychology, 10:2905.

show abstract

Commonsense Interpretation of Triangle Behavior

Cited by 21 publications

References 14 publications

Benchmarks for Automated Commonsense Reasoning: A Survey

Benchmarks for Automated Commonsense Reasoning: A Survey

An Infant-Cognition Inspired Machine Benchmark for Identifying Agency, Affiliation, Belief, and Intention

Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models

Contact Info

Product

Resources

About