Mutual exclusivity as a challenge for deep neural networks

Gandhi, Kanishk; Lake, Brenden M.

doi:10.48550/arxiv.1906.10197

Cited by 4 publications

(4 citation statements)

References 16 publications

(25 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Human infants are not as tabula rasa as models like InferSent but rather encode useful inductive biases (Chomsky & Lightfoot, 2002;Lightfoot & Julia, 1984;Mitchell, 1980;Pearl & Goldwater, 2016;Seidenberg, 1997). Building such biases into our models (Battaglia et al, 2018;Dubey, Agrawal, Pathak, Griffiths, & Efros, 2018;Gandhi & Lake, 2019;Lake et al, 2018) is a promising direction towards scalably learning systematic representations. We also showed how analysis and controlled testing for heuristic strategies in the learning environment can provide rich insights into the representations learned.…”

Section: Discussion and Future Workmentioning

confidence: 99%

Analyzing machine-learned representations: A natural language case study

Dasgupta¹,

Guo²,

Gershman³

et al. 2019

Preprint

View full text Add to dashboard Cite

As modern deep networks become more complex, and get closer to human-like capabilities in certain domains, the question arises of how the representations and decision rules they learn compare to the ones in humans. In this work, we study representations of sentences in one such artificial system for natural language processing. We first present a diagnostic test dataset to examine the degree of abstract composable structure represented. Analyzing performance on these diagnostic tests indicates a lack of systematicity in the representations and decision rules, and reveals a set of heuristic strategies. We then investigate the effect of the training distribution on learning these heuristic strategies, and study changes in these representations with various augmentations to the training set. Our results reveal parallels to the analogous representations in people. We find that these systems can learn abstract rules and generalize them to new contexts under certain circumstances -similar to human zero-shot reasoning. However, we also note some shortcomings in this generalization behavior -similar to human judgment errors like belief bias. Studying these parallels suggests new ways to understand psychological phenomena in humans as well as informs best strategies for building artificial intelligence with human-like language understanding.

show abstract

Section: Discussion and Future Workmentioning

confidence: 99%

Analyzing machine-learned representations: A natural language case study

Dasgupta¹,

Guo²,

Gershman³

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…When children endeavour to learn a new word, they rely on inductive biases to narrow the space of possible meanings: they prefer to predict that the novel word refers to the novel object. However, deep learning algorithms lack this bias [15]. To demonstrate this assumption, we calculate the percentage of novel object in the wrongly labeled vision regions.…”

Section: Zero-shot Learning With Vsepsmentioning

confidence: 99%

Language Models as Zero-shot Visual Semantic Learners

Jiao

Hare

Prügel-Bennett

2021

Preprint

View full text Add to dashboard Cite

Visual Semantic Embedding (VSE) models, which map images into a rich semantic embedding space, have been a milestone in object recognition and zero-shot learning. Current approaches to VSE heavily rely on static word embedding techniques. In this work, we propose a Visual Semantic Embedding Probe (VSEP) designed to probe the semantic information of contextualized word embeddings in visual semantic understanding tasks. We show that the knowledge encoded in transformer language models can be exploited for tasks requiring visual semantic understanding. The VSEP with contextual representations can distinguish word-level object representations in complicated scenes as a compositional zero-shot learner. We further introduce a zero-shot setting with VSEPs to evaluate a model's ability to associate a novel word with a novel visual category. We find that contextual representations in language models outperform static word embeddings, when the compositional chain of object is short. We notice that current visual semantic embedding models lack a mutual exclusivity bias which limits their performance.

show abstract

“…Failure to generalize structurally or failure to produce novel labels? It is known that neural models find it challenging to produce labels they have not seen during training (Gandhi and Lake, 2019). Handling this problem is a necessary part of solving depth generalization, since each of the outputs of the depth generalization cases, such as (5b) below, contains more constants than the training outputs, such as the output of (5a):…”

Section: Lexical Vs Structural Generalizationmentioning

confidence: 99%

COGS: A Compositional Generalization Challenge Based on Semantic Interpretation

Kim

Linzen²

2020

Preprint

View full text Add to dashboard Cite

Natural language is characterized by compositionality: the meaning of a complex expression is constructed from the meanings of its constituent parts. To facilitate the evaluation of the compositional abilities of language processing architectures, we introduce COGS, a semantic parsing dataset based on a fragment of English. The evaluation portion of COGS contains multiple systematic gaps that can only be addressed by compositional generalization; these include new combinations of familiar syntactic structures, or new combinations of familiar words and familiar structures. In experiments with Transformers and LSTMs, we found that in-distribution accuracy on the COGS test set was near-perfect (96-99%), but generalization accuracy was substantially lower (16-35%) and showed high sensitivity to random seed (±6-8%). These findings indicate that contemporary standard NLP models are limited in their compositional generalization capacity, and position COGS as a good way to measure progress.

show abstract

Mutual exclusivity as a challenge for deep neural networks

Cited by 4 publications

References 16 publications

Analyzing machine-learned representations: A natural language case study

Analyzing machine-learned representations: A natural language case study

Language Models as Zero-shot Visual Semantic Learners

COGS: A Compositional Generalization Challenge Based on Semantic Interpretation

Contact Info

Product

Resources

About