ImageTTR: Grounding Type Theory with Records in Image Classification for Visual Question Answering

Abstract

We present ImageTTR, an extension to the Python implementation of Type Theory with Records (pyTTR) which connects formal record type representation with image classifiers implemented as deep neural networks. The Type Theory with Records framework serves as a knowledge representation system for natural language the representations of which are grounded in perceptual information of neural networks. We demonstrate the benefits of this symbolic and data-driven hybrid approach on the task of visual question answering.

What are the Goals of Distributional Semantics?

“…Larsson (2013) represents the meaning of a perceptual concept as a classifier of perceptual input. A number of authors have trained image classifiers using captioned images (for example: Schlangen et al, 2016;Zarrieß and Schlangen, 2017a,b;Utescher, 2019;Matsson et al, 2019).…”