The Flip-flop neuron – A memory efficient alternative for solving challenging sequence processing and decision making problems

Abstract

Sequential decision making tasks that require information integration over extended durations of time are challenging for several reasons including the problem of vanishing gradients, long training times and significant memory requirements. To this end we propose a neuron model fashioned after the JK flip-flops in digital systems. A flip-flop is a sequential device that can store state information of the previous history. We incorporate the JK flip-flop neuron into several deep network architectures and apply the networks to difficult sequence processing problems. The proposed architectures include flip-flop neural networks (FFNNs), bidirectional flip-flop neural networks (BiFFNNs), convolutional flip-flop neural networks (ConvFFNNs), and bidirectional convolutional flip-flop neural networks (BiConvFFNNs). Learning rules of proposed architectures have also been derived. We have considered the most popular benchmark sequential tasks like signal generation, sentiment analysis, handwriting generation, text generation, video frame prediction, lung volume prediction, and action recognition to evaluate the proposed networks. Finally, we compare the results of our networks with the results from analogous networks with Long Short-Term Memory (LSTM) neurons on the same sequential tasks. Our results show that the JK flip-flop networks outperform the LSTM networks significantly or marginally on all the tasks, with only half of the trainable parameters.

Electroencephalogram (EEG) Classification using a bio-inspired Deep Oscillatory Neural Network

“…In future works, for more complex, sequential classification problems (like hybrid behavioral and classifier model, multi-class classification problems etc. ), which involve decision-making over very long sequences, we can add memory elements like flip-flop neurons [76] to our DONN or OCNN models. The study shows the potential application of deep oscillatory neural networks and learning algorithms for classification raw EEG data without feature extraction, which introduces a new perspective to facilitate clinical decision-making.…”

“…Development of a special class of neuron models that may be collectively described as “gating” neuron models - Long Short Term Memory neurons (LSTM) [4], Gated Recurrent Unit (GRU) [4], Flip-flop neuron [48,76] – has immensely enhanced the performance of RNNs [4]. LSTMs and GRUs have shown greater performance over traditional RNNs while classifying sets of emotion from EEG signals [4].…”