Automatic model selection for fully connected neural networks

Frank, Shangjie; Leylaz, Ghazaale; Schütze, Oliver; Sun, Jian‐Qiao

doi:10.1007/s40435-020-00708-w

Cited by 24 publications

(14 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our work, we chose the CNN algorithm to be the model classifying pre- and post-learning SWRs. Indeed, the CNN performed much better, with a greater than 10% average accuracy rate, than other deep learning algorithms, such as fully connected networks 28 , 29 or recurrent neural networks( 30 – 32 ).…”

Section: Methodsmentioning

confidence: 99%

Recognition of post-learning alteration of hippocampal ripples by convolutional neural network differs in the wild-type and AD mice

Hsu

Jura

Shih

et al. 2021

Sci Rep

View full text Add to dashboard Cite

Evidence indicates that sharp-wave ripples (SWRs) are primary network events supporting memory processes. However, some studies demonstrate that even after disruption of awake SWRs the animal can still learn spatial task or that SWRs may be not necessary to establish a cognitive map of the environment. Moreover, we have found recently that despite a deficit of sleep SWRs the APP/PS1 mice, a model of Alzheimer’s disease, show undisturbed spatial reference memory. Searching for a learning-related alteration of SWRs that could account for the efficiency of memory in these mice we use convolutional neural networks (CNN) to discriminate pre- and post-learning 256 ms samples of LFP signals, containing individual SWRs. We found that the fraction of samples that were correctly recognized by CNN in majority of discrimination sessions was equal to ~ 50% in the wild-type (WT) and only 14% in APP/PS1 mice. Moreover, removing signals generated in a close vicinity of SWRs significantly diminished the number of such highly recognizable samples in the WT but not in APP/PS1 group. These results indicate that in WT animals a large subset of SWRs and signals generated in their proximity may contain learning-related information whereas such information seem to be limited in the AD mice.

show abstract

Section: Methodsmentioning

confidence: 99%

Recognition of post-learning alteration of hippocampal ripples by convolutional neural network differs in the wild-type and AD mice

Hsu

Jura

Shih

et al. 2021

Sci Rep

View full text Add to dashboard Cite

show abstract

“…For both fixed and nonfixed topological patterns, the architectural units can be macrounits, which are more commonly known as cells in existing works. In cell-based solution representations [15], [53], [64]- [67], [67]- [69], [71], [79], [159], the encodings of connections among cells follow the ways described in the parts of fixed and nonfixed topological patterns. The cell itself can be treated as a "small" DNN model, which can also be represented in the way following either fixed or nonfixed topological patterns.…”

Section: B Taxonomy and Survey Of Existing Ea-based Approachesmentioning

confidence: 99%

“…For example, in [131], ES is applied to select the learner from Adam and Adadelta and optimize the parameters of the chosen learner. In some works, optimization of learner's parameters is integrated with the model architecture optimization process [51], [53], [55], [127]- [130], [133], [134], [168], [169]. For example, in [127], the EA is applied to design a VGG model, where the parameters of a prespecified model parameter learner are encoded together with the model architecture for solution representations.…”

Section: B Taxonomy and Surveymentioning

confidence: 99%

A Survey on Evolutionary Construction of Deep Neural Networks

Xun

Qin

Gong

et al. 2021

IEEE Trans. Evol. Computat.

View full text Add to dashboard Cite

Automated construction of deep neural networks (DNNs) has become a research hot spot nowadays because DNN's performance is heavily influenced by its architecture and parameters, which are highly task-dependent, but it is notoriously difficult to find the most appropriate DNN in terms of architecture and parameters to best solve a given task. In this work, we provide an insight into the automated DNN construction process by formulating it into a multilevel multiobjective large-scale optimization problem with constraints, where the nonconvex, nondifferentiable, and black-box nature of this problem make evolutionary algorithms (EAs) to stand out as a promising solver. Then, we give a systematical review of existing evolutionary DNN construction techniques from different aspects of this optimization problem and analyze the pros and cons of using EA-based methods in each aspect. This work aims to help DNN researchers to better understand why, where, and how to utilize EAs for automated DNN construction and meanwhile, help EA researchers to better understand the task of automated DNN construction so that they may focus more on EA-favored optimization scenarios to devise more effective techniques.

show abstract

“…Neural architecture search (NAS) is a popular solution to deep learning model selection [57,62,63,102], which automatically selects a good neural network architecture for a given learning task. Since an overly complex model may take too-long training time and thus may become a serious obstacle of neural architecture search [57,62], the accuracy-complexity tradeoff is an important consideration in neural architecture search. Liu et al [62] propose Progressive Neural Architecture Search, which searches for convolutional neural network architectures in the increasing order of model complexity.…”

Section: Model Complexity In Model Selection and Designmentioning

confidence: 99%

Model Complexity of Deep Learning: A Survey

Hu¹,

Chu²,

Pei³

et al. 2021

Preprint

View full text Add to dashboard Cite

Model complexity is a fundamental problem in deep learning. In this paper we conduct a systematic overview of the latest studies on model complexity in deep learning. Model complexity of deep learning can be categorized into expressive capacity and effective model complexity. We review the existing studies on those two categories along four important factors, including model framework, model size, optimization process and data complexity. We also discuss the applications of deep learning model complexity including understanding model generalization capability, model optimization, and model selection and design. We conclude by proposing several interesting future directions.

show abstract

Automatic model selection for fully connected neural networks

Cited by 24 publications

References 29 publications

Recognition of post-learning alteration of hippocampal ripples by convolutional neural network differs in the wild-type and AD mice

Recognition of post-learning alteration of hippocampal ripples by convolutional neural network differs in the wild-type and AD mice

A Survey on Evolutionary Construction of Deep Neural Networks

Model Complexity of Deep Learning: A Survey

Contact Info

Product

Resources

About