A biological mechanism for Bayesian feature selection: Weight decay and raising the LASSO

Connor, Patrick C.; Hollensen, Paul; Krigolson, Olav; Trappenberg, Thomas

doi:10.1016/j.neunet.2015.03.005

Cited by 16 publications

(12 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For the sake of simplicity, we used a raw pixel as the basic feature which could be selected as an informative subset of the input data space. It has been shown that specific forms of weight decay or regularization provide a mechanism for biologically plausible Bayesian feature selection [ 53 – 55 ]. In our ensemble system, selective projections from the input layer to the ensemble WTAs effectively implemented pixel/feature selection in this regard.…”

Section: Resultsmentioning

confidence: 99%

Unsupervised Learning in an Ensemble of Spiking Neural Networks Mediated by ITDP

et al. 2016

View full text Add to dashboard Cite

We propose a biologically plausible architecture for unsupervised ensemble learning in a population of spiking neural network classifiers. A mixture of experts type organisation is shown to be effective, with the individual classifier outputs combined via a gating network whose operation is driven by input timing dependent plasticity (ITDP). The ITDP gating mechanism is based on recent experimental findings. An abstract, analytically tractable model of the ITDP driven ensemble architecture is derived from a logical model based on the probabilities of neural firing events. A detailed analysis of this model provides insights that allow it to be extended into a full, biologically plausible, computational implementation of the architecture which is demonstrated on a visual classification task. The extended model makes use of a style of spiking network, first introduced as a model of cortical microcircuits, that is capable of Bayesian inference, effectively performing expectation maximization. The unsupervised ensemble learning mechanism, based around such spiking expectation maximization (SEM) networks whose combined outputs are mediated by ITDP, is shown to perform the visual classification task well and to generalize to unseen data. The combined ensemble performance is significantly better than that of the individual classifiers, validating the ensemble architecture and learning mechanisms. The properties of the full model are analysed in the light of extensive experiments with the classification task, including an investigation into the influence of different input feature selection schemes and a comparison with a hierarchical STDP based ensemble architecture.

show abstract

Section: Resultsmentioning

confidence: 99%

Unsupervised Learning in an Ensemble of Spiking Neural Networks Mediated by ITDP

et al. 2016

View full text Add to dashboard Cite

show abstract

“…Compared to OLS, whose predicted coefficient is an unbiased estimator of both ridge regression and LASSO sacrifice a little bias in order to reduce the variance of the predicted values and improve the overall prediction accuracy. In this past decade, LASSO has been widely applied in many different ways and variants (Tibshirani et al 2005 ; Colombani et al 2013 ; Yamada et al 2014 ; Toiviainen et al 2014 ; Connor et al 2015 ).…”

Section: Methodsmentioning

confidence: 99%

Improving the prediction of going concern of Taiwanese listed companies using a hybrid of LASSO with data mining techniques

2016

View full text Add to dashboard Cite

The purpose of this study is to establish rigorous and reliable going concern doubt (GCD) prediction models. This study first uses the least absolute shrinkage and selection operator (LASSO) to select variables and then applies data mining techniques to establish prediction models, such as neural network (NN), classification and regression tree (CART), and support vector machine (SVM). The samples of this study include 48 GCD listed companies and 124 NGCD (non-GCD) listed companies from 2002 to 2013 in the TEJ database. We conduct fivefold cross validation in order to identify the prediction accuracy. According to the empirical results, the prediction accuracy of the LASSO–NN model is 88.96 % (Type I error rate is 12.22 %; Type II error rate is 7.50 %), the prediction accuracy of the LASSO–CART model is 88.75 % (Type I error rate is 13.61 %; Type II error rate is 14.17 %), and the prediction accuracy of the LASSO–SVM model is 89.79 % (Type I error rate is 10.00 %; Type II error rate is 15.83 %).

show abstract

“…If a total number of filters are K then a number of biases are K because each filter has a single bias [1], [2], [4], [8], [10], [11], [16]. Additionally, a stride and a pad are given along with a learning rate and weight decay [11], [17], [18].…”

Section: An Open Problem Of Setting Hyperparametersmentioning

confidence: 99%

“…This leads to ultimately overlapping receptive fields between the depth columns, producing ultimately large output volumes. The higher stride makes the receptive fields overlap less and the resulting output volume will have smaller dimensions [10], [11], [15], [17]. Zero-padding is used to exactly preserve the spatial size of output volumes.…”

Section: An Open Problem Of Setting Hyperparametersmentioning

confidence: 99%

Smooth Non-increasing Square Spatial Extents of Filters in Convolutional Layers of CNNs for Image Classification Problems

Romanuke

2018

Applied Computer Systems

View full text Add to dashboard Cite

The present paper considers an open problem of setting hyperparameters for convolutional neural networks aimed at image classification. Since selecting filter spatial extents for convolutional layers is a topical problem, it is approximately solved by accumulating statistics of the neural network performance. The network architecture is taken on the basis of the MNIST database experience. The eight-layered architecture having four convolutional layers is nearly best suitable for classifying small and medium size images. Image databases are formed of grayscale images whose size range is 28 × 28 to 64 × 64 by step 2. Except for the filter spatial extents, the rest of those eight layer hyperparameters are unalterable, and they are chosen scrupulously based on rules of thumb. A sequence of possible filter spatial extents is generated for each size. Then sets of four filter spatial extents producing the best performance are extracted. The rule of this extraction that allows selecting the best filter spatial extents is formalized with two conditions. Mainly, difference between maximal and minimal extents must be as minimal as possible. No unit filter spatial extent is recommended. The secondary condition is that the filter spatial extents should constitute a non-increasing set. Validation on MNIST and CIFAR- 10 databases justifies such a solution, which can be extended for building convolutional neural network classifiers of colour and larger images.

show abstract

A biological mechanism for Bayesian feature selection: Weight decay and raising the LASSO

Cited by 16 publications

References 36 publications

Unsupervised Learning in an Ensemble of Spiking Neural Networks Mediated by ITDP

Unsupervised Learning in an Ensemble of Spiking Neural Networks Mediated by ITDP

Improving the prediction of going concern of Taiwanese listed companies using a hybrid of LASSO with data mining techniques

Smooth Non-increasing Square Spatial Extents of Filters in Convolutional Layers of CNNs for Image Classification Problems

Contact Info

Product

Resources

About