On the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures

Bianchini, Monica; Scarselli, Franco

doi:10.1109/tnnls.2013.2293637

Cited by 453 publications

(267 citation statements)

References 35 publications

Supporting

Mentioning

262

Contrasting

Unclassified

Order By: Relevance

“…It was noted in [6] that the VC dimension proposed for Neural Networks is also applicable to Deep Neural Networks. It was shown in [2] that for neural nets with sigmoidal activation function, the VC-dimension is loosely upper-bounded by O(w 4 ) where w is the number of free parameters in the network.…”

Section: Vc Dimension Of Deep Neural Net-work and Classification Accmentioning

confidence: 99%

A theoretical analysis of Deep Neural Networks for texture classification

Basu

Manohar

Mukhopadhyay

et al. 2016

2016 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

We investigate the use of Deep Neural Networks for the classification of image datasets where texture features are important for generating class-conditional discriminative representations. To this end, we first derive the size of the feature space for some standard textural features extracted from the input dataset and then use the theory of Vapnik-Chervonenkis dimension to show that hand-crafted feature extraction creates low-dimensional representations which help in reducing the overall excess error rate. As a corollary to this analysis, we derive for the first time upper bounds on the VC dimension of Convolutional Neural Network as well as Dropout and Dropconnect networks and the relation between excess error rate of Dropout and Dropconnect networks. The concept of intrinsic dimension is used to validate the intuition that texture-based datasets are inherently higher dimensional as compared to handwritten digits or other object recognition datasets and hence more difficult to be shattered by neural networks. We then derive the mean distance from the centroid to the nearest and farthest sampling points in an n-dimensional manifold and show that the Relative Contrast of the sample data vanishes as dimensionality of the underlying vector space tends to infinity.

show abstract

Section: Vc Dimension Of Deep Neural Net-work and Classification Accmentioning

confidence: 99%

A theoretical analysis of Deep Neural Networks for texture classification

Basu

Manohar

Mukhopadhyay

et al. 2016

2016 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

show abstract

“…RobERtmimics human recognition of spectroscopic features by using a pre-trained, deep-belief neural network (Hinton 2006(Hinton , 2007Bengio et al 2007;Le Roux & Bengio 2010;Montavon et al 2012;Bianchini & Scarselli 2014) at its core. DBNs are multi-layer, nonlinear transformations of the input data, the emission spectrum in this case, where each consecutive layer presents a progressively higher level of abstraction of the underlying features in the spectrum.…”

Section: Robertmentioning

confidence: 99%

Dreaming of Atmospheres

Waldmann

2016

ApJ

View full text Add to dashboard Cite

Here, we introduce the RobERt(Robotic Exoplanet Recognition) algorithm for the classification of exoplanetary emission spectra. Spectral retrieval of exoplanetary atmospheres frequently requires the preselection of molecular/ atomic opacities to be defined by the user. In the era of open-source, automated, and self-sufficient retrieval algorithms, manual input should be avoided. User dependent input could, in worst-case scenarios, lead to incomplete models and biases in the retrieval. The RobERtalgorithm is based on deep-belief neural (DBN) networks trained to accurately recognize molecular signatures for a wide range of planets, atmospheric thermal profiles, and compositions. Reconstructions of the learned features, also referred to as the "dreams" of the network, indicate good convergence and an accurate representation of molecular features in the DBN. Using these deep neural networks, we work toward retrieval algorithms that themselves understand the nature of the observed spectra, are able to learn from current and past data, and make sensible qualitative preselections of atmospheric opacities to be used for the quantitative stage of the retrieval process.

show abstract

“…For some algorithms, a formal time complexity analysis has been performed [23,8,4,18]. This analysis can be used in choosing an algorithm for a prediction.…”

Section: Time Complexity Of ML Algorithmsmentioning

confidence: 99%

An Investigation on Online Versus Batch Learning in Predicting User Behaviour

Burlutskiy

Petridis

Fish

et al. 2016

Research and Development in Intelligent Systems XXXIII

View full text Add to dashboard Cite

An investigation on how to produce a fast and accurate prediction of user behaviour on the Web is conducted. First, the problem of predicting user behaviour as a classification task is formulated and then the main problems of such real-time predictions are specified: the accuracy and time complexity of the prediction. Second, a method for comparison of online and batch (offline) algorithms used for user behaviour prediction is proposed. Last, the performance of these algorithms using the data from a popular question and answer platform, Stack Overflow, is empirically explored. It is demonstrated that a simple online learning algorithm outperforms state-of-the-art batch algorithms and performs as well as a deep learning algorithm, Deep Belief Networks. The proposed method for comparison of online and offline algorithms as well as the provided experimental evidence can be used for choosing a machine learning set-up for predicting user behaviour on the Web in scenarios where the accuracy and the time performance are of main concern.

show abstract

On the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures

Cited by 453 publications

References 35 publications

A theoretical analysis of Deep Neural Networks for texture classification

A theoretical analysis of Deep Neural Networks for texture classification

Dreaming of Atmospheres

An Investigation on Online Versus Batch Learning in Predicting User Behaviour

Contact Info

Product

Resources

About