Integrating Space, Time, and Orientation in Spiking Neural Networks: A Case Study on Multimodal Brain Data Modeling

Abstract

Recent progress in a noninvasive brain data sampling technology has facilitated simultaneous sampling of multiple modalities of brain data, such as functional magnetic resonance imaging, electroencephalography, diffusion tensor imaging, and so on. In spite of the potential benefits from integrating predictive modeling of multiple modality brain data, this area of research remains mostly unexplored due to a lack of methodological advancements. The difficulty in fusing multiple modalities of brain data within a single model lies in the heterogeneous temporal and spatial characteristics of the data sources. Recent advances in spiking neural network systems, however, provide the flexibility to incorporate multidimensional information within the model. This paper proposes a novel, unsupervised learning algorithm for fusing temporal, spatial, and orientation information in a spiking neural network architecture that could potentially be used to understand and perform predictive modeling using multimodal data. The proposed algorithm is evaluated both qualitatively and quantitatively using synthetically generated data to characterize its behavior and its ability to utilize spatial, temporal, and orientation information within the model. This leads to improved pattern recognition capabilities and performance along with robust interpretability of the brain data. Furthermore, a case study is presented, which aims to build a computational model that discriminates between people with schizophrenia who respond or do not respond to monotherapy with the antipsychotic clozapine.

Accurate and Efficient LIF-Nets for 3D Detection and Recognition

“…Lately, SNNs have shown a competitive performance compared with Artificial Neural Networks (ANNs) with the recent developments of improved training algorithms [21]- [23], and have been applied in Electroencephalogram (EEG) brain data [24], [25] and functional Magnetic Resonance Imagining (fMRI) for spatial-temporal cognitive processes [26], [27]. In computer vision tasks, SNNs have achieved outstanding performance in N-MNIST dataset and Cifar10-DVS dataset for image classification and dynamic visual recognition [28]- [30].…”

Construction of Knowledge Graph of 3D Clothing Design Resources Based on Multimodal Clustering Network

“…The iso-lines were evenly segmented, and each subpoint was connected according to a certain topological relationship to obtain the clothing surface; then find out the key points of the 3D clothing outline position of the clothing from the segmentation points to form the feature control lines and realize the deformation of the surface through them [ 19 ]. On the surface of the 3D clothing resource model, the new surface obtained by moving the surface to the normal direction for a certain distance is used as the clothing surface, so as to design tight clothing, so as to realize free drawing of 3D curves on the surface of 3D clothing resources and then use these curves as boundaries to obtain clothing surfaces through interpolation [ 20 – 22 ]. This method is flexible and free, but due to the complex control of 2D to 3D conversion, it can only realize the design of simple clothing surfaces [ 23 – 26 ].…”

Spiking Neural Networks for Computational Intelligence: An Overview

“…-Integrating time, space and orientation data, such as fMRI and DTI [66,69]: An extension of the STDP learning rule was proposed in [69], called oiSTDP, where if two or more postsynaptic neurons spike after a pre-synaptic neuron, the closer a postsynaptic neuron is to the orientation vector, the higher the increase is in the connection weight of that postsynaptic neuron. The proposed rules are utilized for integrating MRI and DTI data to create a personalized model for predicting the response of schizophrenic patient to clozapine.…”