Statistical Mechanics of Learning

Abstract

Preface page ix 1 Getting Started 1.1 Artificial neural networks 1.2 A simple example 1.3 General setup 1.4 Problems 2 Perceptron Learning-Basics 2.1 Gibbs learning 2.2 The annealed approximation 2.3 The Gardner analysis 2.4 Summary 2.5 Problems 3 A Choice of Learning Rules 3.1 The Hebb rule 3.2 The perceptron rule 3.3 The pseudo-inverse rule 3.4 The adaline rule 3.5 Maximal stability 3.6 The Bayes rule 3.7 Summary 3.8 Problems 4 Augmented Statistical Mechanics Formulation 4.1 Maximal stabilities 4.2 Gibbs learning at non-zero temperature 4.3 General statistical mechanics formulation 4.4 Learning rules revisited 4.5 The optimal potential 4.

Learning dynamics and robustness of vector quantization and neural gas

“…Following the lines of the theory of on-line learning, e.g. [8], in the thermodynamic limit N → ∞ the system can be fully described in terms of a few characteristic quantities, or so-called order parameters. A suitable set of characteristic quantities for the considered learning model is:…”

“…(8). While most of the integrations can be performed analytically, some have to be implemented numerically.…”

“…Methods from statistical physics and the theory of on-line learning [8] allow for an exact mathematical description of learning systems for high dimensional data. In the limit of infinite dimensionality, such systems can be fully described in terms of a few characteristic quantities, the so-called order parameters.…”

“…By integrating these ODEs, it is possible to analyse the performance of VQ algorithms in terms of stability, sensitivity to initial conditions, and achievable quantization error. This successful approach has also been reviewed in [8,16], among others. The extension of the theoretical analysis of simple (WTA-based) vector quantization with two prototypes and two clusters introduced in an earlier work [5] is not straightforward.…”

Methods and Techniques of Complex Systems Science: An Overview

“…The application of VC theory to them is quite well-advanced [34,35], but there are many other approaches, including ones based on statistical mechanics [36]. It is notoriously hard to understand why they make the predictions they do.…”