The Lottery Ticket Hypothesis for Self-attention in Convolutional Neural Network

Zhongzhan, Huang,; Liang, Senwei; Liang, Mingfu; Wang, He; Yang, Haizhao; Li, Lin

doi:10.48550/arxiv.2207.07858

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The CNN was employed to extract spatial information from image frames and the BiLSTM module was applied to capture temporal dependencies between features extracted from different frames. Inspired by the ability of attention mechanism to enhance the model generalization [10][11][12][13] , a temporal attention module with Bahdanau's additive style 14 was added to enhance the temporal learning of the calcium dynamics.…”

Section: Introductionmentioning

confidence: 99%

Attention-based CNN-BiLSTM for sleep state classification of spatiotemporal wide-field calcium imaging data

Zhang¹,

Landsness²,

Culver³

et al. 2023

Neural Imaging and Sensing 2023

View full text Add to dashboard Cite

Wide-field calcium imaging (WFCI) with genetically encoded calcium indicators allows spatiotemporal recordings of neuronal activity in preclinical models. When applied to the study of sleep, WFCI data are manually scored into sleep states of wakefulness, non-rapid eye movement (NREM) and REM by use of adjunct electroencephalogram (EEG) and electromyogram (EMG) recordings. However, this process is time-consuming, invasive and suffers from low inter-and intra-rater reliability. To overcome these limitations, an automated sleep state classification method that operates on spatiotemporal WFCI recordings is desired. Previous work that classifies sleep states from WFCI data by use of multiplex visibility graphs and deep learning only leverages shared information derived from average time series across parcellated brain regions, and thus fails to fully explore the spatiotemporal calcium dynamics recorded. In this work, a hybrid network architecture consisting of a convolutional neural network (CNN) to extract spatial features of image frames and a bidirectional long short-term memory network (BiLSTM) with attention mechanism to identify temporal dependencies among different time points was proposed to jointly learn spatial and temporal information from the WFCI sleep data. Nineteen transgenic mice expressing GCaMP6f in excitatory neurons were used for network training and testing. The CNN-BiLSTM achieved a weighted F1-score of 0.84 and Cohen's κ of 0.64, indicating substantial agreement with EEG/EMG-based human scoring. The gradient-weighted class activation maps were computed to provide deeper insights into the brain regions most relevant to the inference of individual sleep state. This work will enable further investigation of sleep neural activity using WFCI.

show abstract

Section: Introductionmentioning

confidence: 99%

Attention-based CNN-BiLSTM for sleep state classification of spatiotemporal wide-field calcium imaging data

Zhang¹,

Landsness²,

Culver³

et al. 2023

Neural Imaging and Sensing 2023

View full text Add to dashboard Cite

show abstract

“…Neural Networks [12], Convolutional Neural Networks [13], Graph Convolution Neural Networks, and Attention Convolutional Neural Networks [14]), our method significantly reduced the computation time and achieved a precision of about 80%. Importantly, we provide an algorithm that balanced computation time and precisions, so that we can further explore the biological significance represented by the cavities.…”

Section: Introductionmentioning

confidence: 99%

Efficient Cavity Searching for Gene Network of Influenza A Virus

Li¹,

Zhao²,

Su³

et al. 2022

Preprint

View full text Add to dashboard Cite

High order structures (cavities and cliques) of the gene network of influenza A virus reveal tight associations among viruses during evolution and are key signals that indicate viral cross-species infection and cause pandemics. As indicators for sensing the dynamic changes of viral genes, these higher order structures have been the focus of attention in the field of virology. However, the size of the viral gene network is usually huge, and searching these structures in the networks introduces unacceptable delay. To mitigate this issue, in this paper, we propose a simple-yet-effective model named Hy-perSearch based on deep learning to search cavities in a computable complex network for influenza virus genetics. Extensive experiments conducted on a public influenza virus dataset demonstrate the effectiveness of HyperSearch over other advanced deep-learning methods without any elaborated model crafting. Moreover, HyperSearch can finish the search works in minutes while 0-1 programming takes days. Since the proposed method is simple and easy to be transferred to other complex networks, HyperSearch has the potential to facilitate the monitoring of dynamic changes in viral genes and help humans keep up with the pace of virus mutations.

show abstract

The Lottery Ticket Hypothesis for Self-attention in Convolutional Neural Network

Cited by 2 publications

References 0 publications

Attention-based CNN-BiLSTM for sleep state classification of spatiotemporal wide-field calcium imaging data

Attention-based CNN-BiLSTM for sleep state classification of spatiotemporal wide-field calcium imaging data

Efficient Cavity Searching for Gene Network of Influenza A Virus

Contact Info

Product

Resources

About