Analysis of Liquid Ensembles for Enhancing the Performance and Accuracy of Liquid State Machines

Abstract

Liquid state machine (LSM), a bio-inspired computing model consisting of the input sparsely connected to a randomly interlinked reservoir (or liquid) of spiking neurons followed by a readout layer, finds utility in a range of applications varying from robot control and sequence generation to action, speech, and image recognition. LSMs stand out among other Recurrent Neural Network (RNN) architectures due to their simplistic structure and lower training complexity. Plethora of recent efforts have been focused toward mimicking certain characteristics of biological systems to enhance the performance of modern artificial neural networks. It has been shown that biological neurons are more likely to be connected to other neurons in the close proximity, and tend to be disconnected as the neurons are spatially far apart. Inspired by this, we propose a group of locally connected neuron reservoirs, or an ensemble of liquids approach, for LSMs. We analyze how the segmentation of a single large liquid to create an ensemble of multiple smaller liquids affects the latency and accuracy of an LSM. In our analysis, we quantify the ability of the proposed ensemble approach to provide an improved representation of the input using the Separation Property (SP) and Approximation Property (AP). Our results illustrate that the ensemble approach enhances class discrimination (quantified as the ratio between the SP and AP), leading to better accuracy in speech and image recognition tasks, when compared to a single large liquid. Furthermore, we obtain performance benefits in terms of improved inference time and reduced memory requirements, due to lowered number of connections and the freedom to parallelize the liquid evaluation process.

Biologically Plausible Class Discrimination Based Recurrent Neural Network Training for Motor Pattern Generation

“…Therefore, we use insights from linear discriminant analysis (LDA) (Fisher, 1936 ; Fukunaga and Mantock, 1983 ; Hourdakis and Trahanias, 2013 ) to quantify the separation between the class attractors. The between class scatter matrix in the following equation contains information on how far each data point is located from the global mean, in the high dimensional space (Fukunaga and Mantock, 1983 ; Wijesinghe et al, 2019 ). Each data point is a vector that contains all the elements in an attractor matrix.…”

“…In the equation, μ i is the sample mean vector (centroid) of class ω i , P (ω i ) is the probability of class ω i , L is the number of classes, and μ g is the global sample mean vector. The single measure that quantifies the separation is given by the trace of the above matrix (Wijesinghe et al, 2019 ).…”

“…In a reservoir with randomly initialized weights (i.e., when no learning is involved to tune the connections), it is a good practice to have sparsity within the network in order to get better projections of the inputs. For example, multiple sparsely connected small reservoirs can give better class discrimination (hence better accuracy) for spatio-temporal data classification tasks using reservoir computing (Wijesinghe et al, 2019 ). This is due to the fact that different combinations from the same set of inputs were fed to the readout by means of an ensemble of reservoirs.…”

Neural Network for Low-Memory IoT Devices and MNIST Image Recognition Using Kernels Based on Logistic Map

“…Nevertheless, in a number of cases, for example, for r = 1.65, 1.805, local maxima of classification accuracy arise at the border of strong chaos and the ordered behavior of the logistic mapping. The important role of the chaotic dynamics of the reservoir was highlighted in the studies on ESNs (setting the spectral radius) [29,45], and LSMs (setting separation property) [30]. A detailed study of the influences of the chaos parameters of the logistic mapping on the classification accuracy of the LogNNet neural network can be a topic for future research.…”

“…An alternative approach to reduce the number of trained weights is based on physical reservoir computing (RC) [23]. RC uses complex physical dynamic systems (coupled oscillators [24][25][26], memristor crossbar arrays [27], opto-electronic feedback loop [28]), or recurrent neural networks (echo state networks (ESNs) [29] and liquid state machines (LSMs) [30]), as reservoirs with rich dynamics and powerful computing capabilities. The couplings in the reservoir are not trained, but are specified in a special way.…”