Neural Module Networks

Andreas, Jacob; Rohrbach, Marcus; Darrell, Trevor; Klein, Dan

doi:10.1109/cvpr.2016.12

Cited by 830 publications

(801 citation statements)

References 23 publications

Supporting

Mentioning

800

Contrasting

Order By: Relevance

“…In contrast to VQA, we use synthetic images and emphasize representing a broad range of language phenomena. Our motivation is similar to that of SHAPES (Andreas et al, 2016b) and CLEVR (Johnson et al, 2016). Both datasets also use synthetic images and emphasize representing diverse spatial language.…”

Section: Related Work and Datasetsmentioning

confidence: 99%

“…NMN The neural module networks approach of Andreas et al (2016b). We experiment with the default maximum leaves of two, and with allowing for more expressive representations with a maximum leaves of five.…”

Section: Image Representationmentioning

confidence: 99%

“…VQA (Antol et al, 2015), while lexically and visually diverse, includes relatively short sentences with limited coverage of such phenomena. CLEVR (Johnson et al, 2016) and SHAPES (Andreas et al, 2016b), in contrast, display complex compositional structure, but include only synthetic language.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Corpus of Natural Language for Visual Reasoning

Suhr¹,

Lewis²,

Yeh³

et al. 2017

Proceedings of the 55th Annual Meeting of the Association For Computational Linguistics (Volume 2: Short Papers)

163

157

View full text Add to dashboard Cite

We present a new visual reasoning language dataset, containing 92,244 pairs of examples of natural statements grounded in synthetic images with 3,962 unique sentences. We describe a method of crowdsourcing linguistically-diverse data, and present an analysis of our data. The data demonstrates a broad set of linguistic phenomena, requiring visual and set-theoretic reasoning. We experiment with various models, and show the data presents a strong challenge for future research.

show abstract

Section: Related Work and Datasetsmentioning

confidence: 99%

Section: Image Representationmentioning

confidence: 99%

See 1 more Smart Citation

A Corpus of Natural Language for Visual Reasoning

Suhr¹,

Lewis²,

Yeh³

et al. 2017

Proceedings of the 55th Annual Meeting of the Association For Computational Linguistics (Volume 2: Short Papers)

163

157

View full text Add to dashboard Cite

show abstract

“…Our approach is inspired by recent work in using neural networks to infer programs expressed in some high-level language, e.g., to answer question involving complex arithmetic, logical, or semantic parsing operations [33]- [41]. Approaches, such as [42], [43], produce programs composed of functions that perform compositional reasoning on an image using an execution engine consisting of neural modules [44]. Similarly, our method produces a program consisting of shape modeling instructions to match a target image by incorporating a shape renderer.…”

Section: Neural Program Inductionmentioning

confidence: 99%

CSGNet: Neural Shape Parser for Constructive Solid Geometry

Sharma

Goyal

Liu

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

136

View full text Add to dashboard Cite

Constructive Solid Geometry (CSG) is a geometric modeling technique that defines complex shapes by recursively applying boolean operations on primitives such as spheres and cylinders. We present CSGNET, a deep network architecture that takes as input a 2D or 3D shape and outputs a CSG program that models it. Parsing shapes into CSG programs is desirable as it yields a compact and interpretable generative model. However, the task is challenging since the space of primitives and their combinations can be prohibitively large. CSGNET uses a convolutional encoder and recurrent decoder based on deep networks to map shapes to modeling instructions in a feed-forward manner and is significantly faster than bottom-up approaches. We investigate two architectures for this task -a vanilla encoder (CNN) -decoder (RNN) and another architecture that augments the encoder with an explicit memory module based on the program execution stack. The stack augmentation improves the reconstruction quality of the generated shape and learning efficiency. Our approach is also more effective as a shape primitive detector compared to a state-of-the-art object detector. Finally, we demonstrate CSGNET can be trained on novel datasets without program annotations through policy gradient techniques.

show abstract

“…In this work, to address the limitations of the existing works, we propose a novel NEP (Neural Embedding Propagation) framework for SSL over heterogeneous networks. Figure 1 gives a running toy example of NEP, which is a powerful yet efficient neural framework that coherently combines an object encoder [16], [17] and a modular network [24], [25]. It leverages the compositional nature of metapaths and trivially generalizes to attributed networks.…”

Section: Introductionmentioning

confidence: 99%

Neural Embedding Propagation on Heterogeneous Networks

Yang

Zhang

2019

2019 IEEE International Conference on Data Mining (ICDM)

View full text Add to dashboard Cite

Classification is one of the most important problems in machine learning. To address label scarcity, semi-supervised learning (SSL) has been intensively studied over the past two decades, which mainly leverages data affinity modeled by networks. Label propagation (LP), however, as the most popular SSL technique, mostly only works on homogeneous networks with single-typed simple interactions. In this work, we focus on the more general and powerful heterogeneous networks, which accommodate multi-typed objects and links, and thus endure multi-typed complex interactions. Specifically, we propose neural embedding propagation (NEP), which leverages distributed embeddings to represent objects and dynamically composed modular networks to model their complex interactions. While generalizing LP as a simple instance, NEP is far more powerful in its natural awareness of different types of objects and links, and the ability to automatically capture their important interaction patterns. Further, we develop a series of efficient training strategies for NEP, leading to its easy deployment on real-world heterogeneous networks with millions of objects. With extensive experiments on three datasets, we comprehensively demonstrate the effectiveness, efficiency, and robustness of NEP compared with stateof-the-art network embedding and SSL algorithms.

show abstract

Neural Module Networks

Cited by 830 publications

References 23 publications

A Corpus of Natural Language for Visual Reasoning

A Corpus of Natural Language for Visual Reasoning

CSGNet: Neural Shape Parser for Constructive Solid Geometry

Neural Embedding Propagation on Heterogeneous Networks

Contact Info

Product

Resources

About