From spatial navigation via visual construction to episodic memory and imagination

Abstract

This hybrid of review and personal essay argues that models of visual construction are essential to extend spatial navigation models to models that link episodic memory and imagination. The starting point is the TAM-WG model, combining the Taxon Affordance Model and the World Graph model of spatial navigation. The key here is to reject approaches in which memory is restricted to unanalyzed views from familiar places, and their later recall. Instead, we will seek mechanisms for imagining truly novel scenes and episodes. We thus introduce a specific variant of schema theory and VISIONS, a cooperative computation model of visual scene understanding in which a scene is represented by an assemblage of schema instances with links to lower-level "patches" of relevant visual data. We sketch a new conceptual framework for future modeling, Visual Integration of Diverse Multi-Modal Aspects, by extending VISIONS from static scenes to episodes combining agents, actions and objects and assess its relevance to both navigation and episodic memory. We can then analyze imagination as a constructive process that combines aspects of memories of prior episodes along with other schemas and adjusts them into a coherent whole which, through expectations associated with diverse episodes and schemas, may yield the linkage of episodes that constitutes a dream or a narrative. The result is IBSEN, a conceptual model of Imagination in Brain Systems for Episodes and Navigation. The essay closes by analyzing other papers in this Special Issue to assess to what extent their results relate to the research proposed here.

Exploratory exploitation and exploitative exploration: The phenomenology of play and the computational dynamics of search

“…Recent work (Arbib, 2020(Arbib, , 2021 analyzes how architects design buildings, constructing new patterns in memory (in diverse brains and the external representations) that cumulatively yield a plan for a new building. The architect does not manipulate an extant semantic network but instead creates rich "mental constructions" that can guide the physical construction of buildingsdevising spaces and shaping and relating forms to serve stipulated functions, be aesthetically pleasing, conform with the site, and be built with available funds.…”

“…Designers exploit their own diverse forms of long-term memoryepisodic, procedural, and semanticto design spaces that will structure the experience of the building's users as they develop behaviors that, at least in part, emerge as variations on the scripts (patterns of behavior that can be adapted to varied circumstances) the architect imagined. Our modeling (Arbib, 2020(Arbib, , 2021 extends navigation to controlling transitions between affordances (opportunities for action) in supporting the constraints of scripts. The architect must transform each script into design ideas for a WG linking places that need to be included in a building to satisfy the script.…”

The creativity of architects

“…Recent work (Arbib, 2020, 2021) analyzes how architects design buildings, constructing new patterns in memory (in diverse brains and the external representations) that cumulatively yield a plan for a new building. The architect does not manipulate an extant semantic network but instead creates rich “mental constructions” that can guide the physical construction of buildings – devising spaces and shaping and relating forms to serve stipulated functions, be aesthetically pleasing, conform with the site, and be built with available funds.…”

Neurorobots as a Means Toward Neuroethology and Explainable AI

“…For instance, a robot might be able to recall past episodes of its experiences with handling an object or visiting a location to explain its current actions. Robot experiments allow us to test such models that link perception, memory, cognitive mapping, and spatial navigation (Arbib, 2020).…”