Self-organizing neural networks for universal learning and multimodal memory encoding

Tan, Ah-Hwee; Subagdja, Budhitama; Wang, Di; Meng, Lei

doi:10.1016/j.neunet.2019.08.020

Cited by 31 publications

(15 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The higher accuracy is achieved thanks to a redundancy gain that reduces the amount of uncertainty in the resulting information. Recent works show a growing interest toward multi-sensory fusion in several application areas, such as developmental robotics (Droniou et al, 2015 ; Zahra and Navarro-Alarcon, 2019 ), audio-visual signal processing (Shivappa et al, 2010 ; Rivet et al, 2014 ), spatial perception (Pitti et al, 2012 ), attention-driven selection (Braun et al, 2019 ) and tracking (Zhao and Zeng, 2019 ), memory encoding (Tan et al, 2019 ), emotion recognition (Zhang et al, 2019 ), multi-sensory classification (Cholet et al, 2019 ), HMI (Turk, 2014 ), remote sensing and earth observation (Debes et al, 2014 ), medical diagnosis (Hoeks et al, 2011 ), and understanding brain functionality (Horwitz and Poeppel, 2002 ).…”

Section: Introductionmentioning

confidence: 99%

Hand-Gesture Recognition Based on EMG and Event-Based Camera Sensor Fusion: A Benchmark in Neuromorphic Computing

et al. 2020

View full text Add to dashboard Cite

Hand gestures are a form of non-verbal communication used by individuals in conjunction with speech to communicate. Nowadays, with the increasing use of technology, hand-gesture recognition is considered to be an important aspect of Human-Machine Interaction (HMI), allowing the machine to capture and interpret the user's intent and to respond accordingly. The ability to discriminate between human gestures can help in several applications, such as assisted living, healthcare, neuro-rehabilitation, and sports. Recently, multi-sensor data fusion mechanisms have been investigated to improve discrimination accuracy. In this paper, we present a sensor fusion framework that integrates complementary systems: the electromyography (EMG) signal from muscles and visual information. This multi-sensor approach, while improving accuracy and robustness, introduces the disadvantage of high computational cost, which grows exponentially with the number of sensors and the number of measurements. Furthermore, this huge amount of data to process can affect the classification latency which can be crucial in real-case scenarios, such as prosthetic control. Neuromorphic technologies can be deployed to overcome these limitations since they allow real-time processing in parallel at low power consumption. In this paper, we present a fully neuromorphic sensor fusion approach for hand-gesture recognition comprised of an event-based vision sensor and three different neuromorphic processors. In particular, we used the event-based camera, called DVS, and two neuromorphic platforms, Loihi and ODIN + MorphIC. The EMG signals were recorded using traditional electrodes and then converted into spikes to be fed into the chips. We collected a dataset of five gestures from sign language where visual and electromyography signals are synchronized. We compared a fully neuromorphic approach to a baseline implemented using traditional machine learning approaches on a portable GPU system. According to the chip's constraints, we designed specific spiking neural networks (SNNs) for sensor fusion that showed classification accuracy comparable to the software baseline. These neuromorphic alternatives have increased inference time, between 20 and 40%, Ceolini et al. Sensor Fusion Neuromorphic Benchmark with respect to the GPU system but have a significantly smaller energy-delay product (EDP) which makes them between 30× and 600× more efficient. The proposed work represents a new benchmark that moves neuromorphic computing toward a real-world scenario.

show abstract

Section: Introductionmentioning

confidence: 99%

Hand-Gesture Recognition Based on EMG and Event-Based Camera Sensor Fusion: A Benchmark in Neuromorphic Computing

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Density cluster: findClusters function in densityClust package [46] Hierarchical cluster: hclust function Som (self-organized map) [47]…”

Section: Data Availabilitymentioning

confidence: 99%

Single-cell transcriptome profiling simulation reveals the impact of sequencing parameters and algorithms on clustering

Liu

Shi

et al. 2021

Preprint

View full text Add to dashboard Cite

Despite of scRNA-seq analytic algorithms developed, their performance for cell clustering cannot be quantified due to the unknown “true” clusters. Referencing the transcriptomic heterogeneity of cell clusters, a “true” mRNA number matrix of cell individuals was defined as ground truth. Based on the matrix and real data generation procedure, a simulation program (SSCRNA) for raw data was developed. Subsequently, the consistence between simulated data and real data was evaluated. Furthermore, the impact of sequencing depth, and algorithms for analyses on cluster accuracy was quantified. As a result, the simulation result is highly consistent with that of the real data. It is found that mis-classification rate can be attributed to multiple reasons on current scRNA platforms, and clustering accuracy is not only sensitive to sequencing depth increasement, but can also be reflected by the position of the cluster on TSNE plot. Among the clustering algorithms, Gaussian normalization method is more appropriate for current workflows. In the clustering algorithms, k-means&louvain clustering method performs better in dimension reduced data than full data, while k-means clustering method is stable under both situations. In conclusion, the scRNA simulation algorithm developed restores the real data generation process, discovered impact of parameters on mis-clustering, compared the normalization/clustering algorithms and provided novel insight into scRNA analyses.

show abstract

“…Multimodal data fusion is thus a direct consequence of the well-accepted paradigm that certain natural processes and phenomena are expressed under completely different physical guises [6]. Recent works show a growing interest toward multimodal association in several applicative areas such as developmental robotics [3], audio-visual signal processing [7,8], spatial perception [9,10], attention-driven selection [11] and tracking [12], memory encoding [13], emotion recognition [14], human-machine interaction [15], remote sensing and earth observation [16], medical diagnosis [17], understanding brain functionality [18], and so forth. Interestingly, the last mentioned application is our starting bloc: how does the brain handle multimodal learning in the natural environment?…”

Section: Introductionmentioning

confidence: 99%

Brain-Inspired Self-Organization with Cellular Neuromorphic Computing for Multimodal Unsupervised Learning

2020

View full text Add to dashboard Cite

Cortical plasticity is one of the main features that enable our ability to learn and adapt in our environment. Indeed, the cerebral cortex self-organizes itself through structural and synaptic plasticity mechanisms that are very likely at the basis of an extremely interesting characteristic of the human brain development: the multimodal association. In spite of the diversity of the sensory modalities, like sight, sound and touch, the brain arrives at the same concepts (convergence). Moreover, biological observations show that one modality can activate the internal representation of another modality when both are correlated (divergence). In this work, we propose the Reentrant Self-Organizing Map (ReSOM), a brain-inspired neural system based on the reentry theory using Self-Organizing Maps and Hebbian-like learning. We propose and compare different computational methods for unsupervised learning and inference, then quantify the gain of the ReSOM in a multimodal classification task. The divergence mechanism is used to label one modality based on the other, while the convergence mechanism is used to improve the overall accuracy of the system. We perform our experiments on a constructed written/spoken digits database and a Dynamic Vision Sensor (DVS)/EletroMyoGraphy (EMG) hand gestures database. The proposed model is implemented on a cellular neuromorphic architecture that enables distributed computing with local connectivity. We show the gain of the so-called hardware plasticity induced by the ReSOM, where the system’s topology is not fixed by the user but learned along the system’s experience through self-organization.

show abstract

Self-organizing neural networks for universal learning and multimodal memory encoding

Cited by 31 publications

References 36 publications

Hand-Gesture Recognition Based on EMG and Event-Based Camera Sensor Fusion: A Benchmark in Neuromorphic Computing

Hand-Gesture Recognition Based on EMG and Event-Based Camera Sensor Fusion: A Benchmark in Neuromorphic Computing

Single-cell transcriptome profiling simulation reveals the impact of sequencing parameters and algorithms on clustering

Brain-Inspired Self-Organization with Cellular Neuromorphic Computing for Multimodal Unsupervised Learning

Contact Info

Product

Resources

About