Endotaxis: A neuromorphic algorithm for mapping, goal-learning, navigation, and patrolling

Abstract

An animal entering a new environment typically faces three challenges: explore the space for resources, memorize their locations, and navigate towards those targets as needed. Experimental work on exploration, mapping, and navigation has mostly focused on simple environments - such as an open arena, a pond [1], or a desert [2] - and much has been learned about neural signals in diverse brain areas under these conditions [3,4]. However, many natural environments are highly constrained, such as a system of burrows, or of paths through the underbrush. More generally, many cognitive tasks are equally constrained, allowing only a small set of actions at any given stage in the process. Here we propose an algorithm that learns the structure of an arbitrary environment, discovers useful targets during exploration, and navigates back to those targets by the shortest path. It makes use of a behavioral module common to all motile animals, namely the ability to follow an odor to its source [5]. We show how the brain can learn to generate internal "virtual odors'" that guide the animal to any location of interest. This endotaxis algorithm can be implemented with a simple 3-layer neural circuit using only biologically realistic structures and learning rules. Several neural components of this scheme are found in brains from insects to humans. Nature may have evolved a general mechanism for search and navigation on the ancient backbone of chemotaxis.

Neural learning rules for generating flexible predictions and computing the successor representation

“…Typically, for purposes of fitting neural data or for RL simulations, γ will take on values as high as 0.9 [18, 49]. However, previous work that used RNN models reported that recurrent dynamics become unstable if the gain γ exceeds a critical value [42, 45]. This could be problematic as we show analytically that the RNN-S update rule is effective only when the network dynamics are stable and do not have non-normal amplification (Supplementary Notes 2).…”

“…Because equivalence is only reached in the asymptotic limit of learning (i.e., ∆J → 0), our RNN-S model learns the SR slowly. In contrast, animals are thought to be able to learn 123 the structure of an environment quickly [45], and neural representations in an environment can also develop quickly [46,47,48]. To remedy this, we introduce a dynamic learning rate that allows for faster normalization of the synaptic weight matrix, similar to the formula for calculating a moving average (Supplementary Notes 4).…”

“…The endotaxis paper [45] uses Oja's rule in an RNN with place-like inputs. The SR can also be learned with an Oja-like learning rule.…”

“…The learning rule and architecture of our model is similar to a hypothesized “endotaxis” model [45]. In the endotaxis model, neurons fire most strongly near a reward, allowing the animal to navigate up a gradient of neural activity akin to navigating up an odor gradient.…”

Seeking entropy: complex behavior from intrinsic motivation to occupy action-state path space

“…Indeed, expressing excessive emphasis on those goals is a sign of psychological disorders [16,17]. Further, setting a reward function by design as the goal of intelligent agents is more often than not arbitrary [14,18,19], leading to the recurrent problem faced by theories of reward maximization of defining what rewards are [20][21][22][23][24]. In some cases, like in artificial games, rewards can be unambiguously defined, such as number of collected points or wins [25].…”