A Bio-Inspired Incremental Learning Architecture for Applied Perceptual Problems

Abstract

We present a biologically inspired architecture for incremental learning that remains resource-efficient even in the face of very high data dimensionalities (>1000) that are typically associated with perceptual problems. In particular, we investigate how a new perceptual (object) class can be added to a trained architecture without retraining, while avoiding the wellknown catastrophic forgetting effects typically associated with such scenarios. At the heart of the presented architecture lies a generative description of the perceptual space by a self-organized approach which at the same time approximates the neighbourhood relations in this space on a two-dimensional plane. This approximation, which closely imitates the topographic organization of the visual cortex, allows an efficient local update rule for incremental learning even in the face of very high dimensionalities, which we demonstrate by tests on the well-known MNIST benchmark. We complement the model by adding a biologically plausible short-term memory system, allowing it to retain excellent classification accuracy even under incremental learning in progress. The short-term memory is additionally used to reinforce new data statistics by replaying previously stored samples during dedicated "sleep" phases.

Computational Advantages of Deep Prototype-Based Learning

“…However to be fair, this parameter search would also have to be performed for convolutional neural network (CNN) and is a property of all deep architectures based on local receptive fields. Given that prototype-based methods in machine learning have a number of highly desirable properties, such as online and incremental learning capacity [7,6], a simple probabilistic interpretation [12] and a natural way of processing multi-class problems, the reduction of resource requirements even when treating complex visual problems seems an important step towards wide-spread use of prototype-based machine learning methods.…”

“…1, we use a prototype-based learning algorithm which is loosely based on the self-organizing map model, see [7]. Inputs are represented by graded neural activities arranged in maps organized on a two-dimensional grid lattice.…”

“…For any map X, the map activity z X ij ∈ Z X at position (i, j) is then derived from the Euclidean distance between the unit prototype w X ij and the current input x: where, as described in [7], g κ (·) is a Gaussian function with an adaptive parameter κ that converts distances into the [0, 1] interval, and f(·) is a monotonous non-linear transfer function, defined as :…”

“…A very popular prototype-based method in computer vision in is particle filtering [5], where a continuous, evolving probability density function is described and updated as a set of prototypes (here denoted particles) whose local density represents local probability density. Prototype-based methods are well suited for incremental learning [6,7] since prototypes have a very obvious interpretation, and can thus be manipulated easily, e.g., by adding, adapting or removing prototypes (see [8] for a precise definition of incremental learning).…”

“…An obvious problem of such flat architectures is the curse of dimensionality: complex probability distributions in high-dimensional spaces may conceivably require a great number of prototypes to be well approximated, so the memory requirements of flat prototype-based learning can become excessive depending on the problem at hand [9]. This study generalizes "flat" prototype-based learning as presented in [7] to a "deep" architecture (see Fig. 1), with localized receptive fields in the lower layers, just as it is the case in convolutional neural networks (CNNs, see [10]).…”