Biologically inspired incremental learning for high-dimensional spaces

Abstract

International audienceWe propose an incremental, highly parallelizable, and constant-time complexity neural learning architecture for multi-class classification (and regression) problems that remains resource-efficient even when the number of input dimensions is very high (≥ 1000). This so-called projection-prediction (PROPRE) architecture is strongly inspired by biological information processing in that it uses a prototype-based, topologically organized hidden layer trained with the SOM learning rule that updates hidden layer weights whenever an error occurs. The SOM learning adapts only the weights of localized neural sub-populations that are similar to the input, which explicitly avoids the catastrophic forgetting effect of MLPs in case new input statistics are presented. The readout layer applies linear regression to hidden layer activities subjected to a transfer function, making the whole system capable of representing strongly non-linear decision boundaries. The resource-efficiency of the algorithm stems from the fact of approximating similarity in the input space by proximity in the SOM layer due to the topological SOM projection property. This avoids the storage of inter-cluster distances (quadratic in number of hidden layer) or input space covariance matrices (quadratic in input dimensions) as K-means, RBF or LWPR would have to do. Tests on the popular MNIST handwritten digit benchmark show that the algorithm compares favorably to state-of-the-art results, and parallelizability is demonstrated by analyzing the efficiency of a parallel GPU implementation of the architecture

A Generative Learning Approach to Sensor Fusion and Change Detection

“…1, to address the proposed fusion problem. It is conceptually similar to several of our previous works in multi-sensory fusion [17], incremental learning [18][19][20] and developmental learning [21,22]. In the light of the evaluation criteria for multi-sensory fusion approaches proposed in Sec.…”

“…This is a prototype-based generative algorithm which therefore permits the detection of outliers and changes in input statistics. Apart from approximating the distribution of input samples by prototype vectors, it implements a topology-preserving mapping of the computed prototypes, which is of no direct concern here, but will become extremely important when incremental learning is concerned [19,18]. As incremental learning is the logical next step following successful change detection, we feel that the use of the SOM model is very well justified in this case.…”

A Bio-Inspired Incremental Learning Architecture for Applied Perceptual Problems

“…• the architecture is conceptually simple, efficient to execute and highly parallelizable as we could show in a previous publication [24] • we present an intuitive way to benchmark incremental learning and apply it to our architecture using two challenging perceptual problems, with excellent results regarding incremental learning per-formance, which never degrades overall precision by more than a few percents.…”

Reducing Catastrophic Forgetting in Self Organizing Maps with Internally-Induced Generative Replay (Student Abstract)

“…Given that units in a competitive learning system like the SOM tend to specialize for certain types of patterns, i.e., data prototypes, such a system appears to be suited for combating catastrophic forgetting (since, in theory, competing units would lead to reduced neural cross-talk, a source of forgetting (French 1999)). Furthermore, approaches, such as (Gepperth et al 2015), have advocated the use of SOMs for continual learning. However, the SOM in its purest form, as we observe in our experiments, is itself prone to forgetting (Mermillod, Bugaiska, and Bonin 2013).…”