Learning to Compose Neural Networks for Question Answering

Andreas, Jacob; Rohrbach, Marcus; Darrell, Trevor; Klein, Dan

doi:10.18653/v1/n16-1181

Cited by 404 publications

(365 citation statements)

References 25 publications

Supporting

Mentioning

364

Contrasting

Order By: Relevance

“…NSM is similar to Neural Programmer (Neelakantan et al, 2015) and Dynamic Neural Module Network (Andreas et al, 2016) in that they all solve the problem of semantic parsing from structured data, and generate programs using similar semantics. The main difference between these approaches is how an intermediate result (the memory) is represented.…”

Section: Related Workmentioning

confidence: 99%

Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision

Chen

Berant

et al. 2017

Proceedings of the 55th Annual Meeting of the Association For Computational Linguistics (Volume 1: Long Papers)

319

298

View full text Add to dashboard Cite

Harnessing the statistical power of neural networks to perform language understanding and symbolic reasoning is difficult, when it requires executing efficient discrete operations against a large knowledge-base. In this work, we introduce a Neural Symbolic Machine (NSM), which contains (a) a neural "programmer", i.e., a sequence-to-sequence model that maps language utterances to programs and utilizes a key-variable memory to handle compositionality (b) a symbolic "computer", i.e., a Lisp interpreter that performs program execution, and helps find good programs by pruning the search space. We apply REINFORCE to directly optimize the task reward of this structured prediction problem. To train with weak supervision and improve the stability of REINFORCE we augment it with an iterative maximum-likelihood training process. NSM outperforms the state-of-theart on the WEBQUESTIONSSP dataset when trained from question-answer pairs only, without requiring any feature engineering or domain-specific knowledge.

show abstract

Section: Related Workmentioning

confidence: 99%

Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision

Chen

Berant

et al. 2017

Proceedings of the 55th Annual Meeting of the Association For Computational Linguistics (Volume 1: Long Papers)

319

298

View full text Add to dashboard Cite

show abstract

“…Components of this pipeline can be trained independently (Sections 5.2 and 5.3) or jointly as a single End-to-End model (Section 5.4). This division of labor also allows for differing amounts of human intervention both during training and in the interpretation of actions and bears some resemblance to (Andreas et al, 2016). Specifically, we will first present results where the model predicts a fixed semantic interpretation of actions which are easily human interpretable (Encoder + Representation).…”

Section: Model Architecturementioning

confidence: 99%

Natural Language Communication with Robots

Bisk

Yüret

Marcu

2016

Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

142

View full text Add to dashboard Cite

We propose a framework for devising empirically testable algorithms for bridging the communication gap between humans and robots. We instantiate our framework in the context of a problem setting in which humans give instructions to robots using unrestricted natural language commands, with instruction sequences being subservient to building complex goal configurations in a blocks world. We show how one can collect meaningful training data and we propose three neural architectures for interpreting contextually grounded natural language commands. The proposed architectures allow us to correctly understand/ground the blocks that the robot should move when instructed by a human who uses unrestricted language. The architectures have more difficulty in correctly understanding/grounding the spatial relations required to place blocks correctly, especially when the blocks are not easily identifiable.

show abstract

“…In particular, different VQA models have focused on how they integrate the question and image inputs in the model. Various VQA techniques were reviewed in [15], where the recent approaches were found to be, − using Bayesian models to exploit the underlying relationships between question-image-answer feature distributions [28], − using the question to break the VQA task into a sequence of modular sub-problems [2]. For example, the question "what is the major class in the image?"…”

Section: Related Workmentioning

confidence: 99%

A Question-Centric Model for Visual Question Answering in Medical Imaging

Löfstedt

Nyholm

et al. 2020

IEEE Trans. Med. Imaging

View full text Add to dashboard Cite

Deep learning methods have proven extremely effective at performing a variety of medical image analysis tasks. With their potential use in clinical routine, their lack of transparency has however been one of their few weak points, raising concerns regarding their behavior and failure modes. While most research to infer model behavior has focused on indirect strategies that estimate prediction uncertainties and visualize model support in the input image space, the ability to explicitly query a prediction model regarding its image content offers a more direct way to determine the behavior of trained models. To this end, we present a novel Visual Question Answering approach that allows an image to be queried by means of a written question. Experiments on a variety of medical and natural image datasets show that by fusing image and question features in a novel way, the proposed approach achieves an equal or higher accuracy compared to current methods.

show abstract

Learning to Compose Neural Networks for Question Answering

Cited by 404 publications

References 25 publications

Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision

Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision

Natural Language Communication with Robots

A Question-Centric Model for Visual Question Answering in Medical Imaging

Contact Info

Product

Resources

About