Statistical physics of learning: Phase transitions in multilayered neural networks

Biehl, Michael; Ahr, M.; Schlösser, E.

doi:10.1007/bfb0108398

Cited by 5 publications

(5 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2, circles), from the Gaussian approximation (Eqs. (7) and (10) and Fig. 2, solid lines), and the exact solution (App.…”

Section: Features Of the Somatic Inputmentioning

confidence: 98%

“…(20)] and thus via the linear field u n [Eq. (14)]. We may therefore define an effective input functionF ¼Fðu n Þ as…”

Section: Deterministic Hopfield Network Of Arborized Neuronsmentioning

confidence: 99%

“…Further, firing-rate models have been developed [11] that reproduce the response properties of detailed models to diverse stimuli and behave like multilayered feed-forward networks of simple-rate neurons [5,8,10]. Two-and multilayer feed-forward networks of binary, deterministic neurons have been studied using statistical physics methods [12][13][14]. In particular, the so-called committee machine may be seen as a neuron model incorporating a layer of dendrites with steplike activation functions, i.e., without analogous signal transmission [15,16].…”

Section: Introduction: Nonadditive Dendritic Input Processing In Nmentioning

confidence: 99%

See 2 more Smart Citations

Statistical Physics of Neural Systems with Nonadditive Dendritic Coupling

2014

View full text Add to dashboard Cite

How neurons process their inputs crucially determines the dynamics of biological and artificial neural networks. In such neural and neural-like systems, synaptic input is typically considered to be merely transmitted linearly or sublinearly by the dendritic compartments. Yet, single-neuron experiments report pronounced supralinear dendritic summation of sufficiently synchronous and spatially close-by inputs. Here, we provide a statistical physics approach to study the impact of such nonadditive dendritic processing on single-neuron responses and the performance of associative-memory tasks in artificial neural networks. First, we compute the effect of random input to a neuron incorporating nonlinear dendrites. This approach is independent of the details of the neuronal dynamics. Second, we use those results to study the impact of dendritic nonlinearities on the network dynamics in a paradigmatic model for associative memory, both numerically and analytically. We find that dendritic nonlinearities maintain network convergence and increase the robustness of memory performance against noise. Interestingly, an intermediate number of dendritic branches is optimal for memory functionality

show abstract

“…2, circles), from the Gaussian approximation (Eqs. (7) and (10) and Fig. 2, solid lines), and the exact solution (App.…”

Section: Features Of the Somatic Inputmentioning

confidence: 98%

“…(20)] and thus via the linear field u n [Eq. (14)]. We may therefore define an effective input functionF ¼Fðu n Þ as…”

Section: Deterministic Hopfield Network Of Arborized Neuronsmentioning

confidence: 99%

Section: Introduction: Nonadditive Dendritic Input Processing In Nmentioning

confidence: 99%

See 1 more Smart Citation

Statistical Physics of Neural Systems with Nonadditive Dendritic Coupling

2014

View full text Add to dashboard Cite

show abstract

“…A topic of particular interest for this work is the analysis of phase transitions in learning processes, i.e. sudden changes of the expected performance with the training set size or other control parameters, see (Kinzel 1998, Opper 1994, Herschkowitz and Opper 2001, Kang et al 1993, Biehl et al 2000, Biehl, Schl össer and Ahr 1998, Ahr et al 1999, Saitta et al 2011) for examples and further references. We systematically study the training of layered networks in student teacher settings similar to the settings in Chapter 2 and 3, see also e.g.…”

Section: Introductionmentioning

confidence: 99%

“…At a critical size of the training set, a favorable configuration with specialized hidden units appears. However, a poorly performing state remains metastable and the specialization required for successful learning can delay the training process significantly (Kang et al 1993, Biehl, Schl össer and Ahr 1998, Ahr et al 1999, Biehl et al 2000.…”

Section: Introductionmentioning

confidence: 99%

Machine learning: statistical physics based theory and smart industry applications

Straat

View full text Add to dashboard Cite

The contributions in this thesis are divided into two main parts: 1) a theoretical analysis of learning in neural networks and Learning Vector Quantization (LVQ) in model situations using statistical physics techniques and 2) the application of machine learning to smart industry settings.In the first part we address highly relevant situations and questions for current machine learning practice: using tools from statistical physics we analyse the learning behaviour in Rectified Linear Unit (ReLU) neural networks and compare it to sigmoidal neural networks in both on-line and off-line supervised learning settings, in order to contribute to the much needed theoretical insight into the properties of the use of the ReLU function, the most popular type of activation function in deep neural networks that are used in many machine learning tasks. Secondly, we analyse neural networks and LVQ under real and virtual concept drift processes that affect many applications of machine learning systems. Our analyses reveal several significant effects, which are, among others: ReLU networks handle overparameterization differently than sigmoidal networks, ReLU networks exhibit favourable second order phase transitions towards hidden unit specialization instead of the first order phase transitions observed for sigmoidal networks and applying weight decay in concept drift scenarios is more effective for ReLU neural networks, in which it significantly accelerates the onset of specialization. For LVQ we find non-trivial dependence of the generalization performance on the learning rate in concept drift situations. Moreover, it is shown that an appropriate amount of weight decay can be beneficial to the performance in the real drift settings. In contrast, the resulting limited flexibility of the prototypes decreases the performance under virtual drift.In the second part of the thesis we focus on the use of computational intelligence approaches in applications in industry. First, we perform a typical Industry 4.0 case study in collaboration with industry that concerns the development of real-time material quality control in a highthroughput production line of steel-based products. In this case study, material measurements iii taken with a fast non-invasive sensor are related to material properties measured by tensile tests. Due to significant correlations between the two types of measurements, we successfully fit and evaluate a model that can estimate material properties in real-time. Furthermore, it is shown on 108 kilometres of processed steel that the model is able to prevent expensive production problems and that it can indicate a risk of the occurrence of production faults.Additionally, we propose a methodology for time series classification that combines Generalized Matrix Learning Vector Quantization (GMLVQ) formulated for complex-valued data with complex-valued Fourier and wavelet features. On several benchmark datasets and a heart beat classification task, learning in the space of complex-valued coefficients is found to yield bett...

show abstract

Empirical Study of Relational Learning Algorithms in the Phase Transition Framework

Alphonse

Osmani

2009

Machine Learning and Knowledge Discovery in Databases

View full text Add to dashboard Cite

Abstract. Relational Learning (RL) has aroused interest to fill the gap between efficient attribute-value learners and growing applications stored in multi-relational databases. However, current systems use generalpurpose problem solvers that do not scale-up well. This is in contrast with the past decade of success in combinatorics communities where studies of random problems, in the phase transition framework, allowed to evaluate and develop better specialised algorithms able to solve real-world applications up to millions of variables. A number of studies have been proposed in RL, like the analysis of the phase transition of a NP-complete sub-problem, the subsumption test, but none has directly studied the phase transition of RL. As RL, in general, is Σ2 − hard, we propose a first random problem generator, which exhibits the phase transition of its decision version, beyond NP. We study the learning cost of several learners on inherently easy and hard instances, and conclude on expected benefits of this new benchmarking tool for RL.

show abstract

Statistical physics of learning: Phase transitions in multilayered neural networks

Cited by 5 publications

References 25 publications

Statistical Physics of Neural Systems with Nonadditive Dendritic Coupling

Statistical Physics of Neural Systems with Nonadditive Dendritic Coupling

Machine learning: statistical physics based theory and smart industry applications

Empirical Study of Relational Learning Algorithms in the Phase Transition Framework

Contact Info

Product

Resources

About