Interfacing language, spatial perception and cognition in Type Theory with Records

Abstract

We argue that computational modelling of perception, action, language, and cognition introduces several requirements of a formal semantic theory and its practical implementations in situated dialogue agents. Using examples of semantic representations of spatial descriptions we show how Type Theory with Records (TTR) satisfies these requirements and provides a promising knowledge representation system for situated agents.

Standpoint semantics for polysemy in spatial prepositions

“…Dobnik and Cooper [35] use type theory with records to provide a computational modelling of spatial descriptions involving perception, movement and natural language. One important feature of their proposal is the ability to deal with low-level perceptual judgements as well as with conceptual judgements.…”

ImageTTR: Grounding Type Theory with Records in Image Classification for Visual Question Answering

“…This section applies the theory outlined above to the case at hand. The main difference against (Dobnik and Cooper, 2017) is that the world is now an image (Equation 8), rather than a 3D point space. A Segment (Equation 9) is now defined as a record type describing a rectangular bounding box within an image.…”

“…Let the world be of some type World , and any portion of the world of some type Segment. These types are left undefined for now, as they differ significantly between (Dobnik and Cooper, 2017) and this project. An object detector function f objdetector : ObjDetector (Equation 3) maps the world to a collection of perceptual objects of the type Obj (Equation 4).…”