From complex to simple : hierarchical free-energy landscape renormalized  in deep neural networks

Yoshino, Hajime

doi:10.21468/scipostphyscore.2.2.005

Cited by 9 publications

(12 citation statements)

References 80 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The former non-rigorousness might be resolved using the techniques developed in analyzing other convex learning problems (Mignacco et al, 2020;Gerbelot et al, 2020). For simplification of the model architecture, we might be able to extend the present analysis based on the previous analytical techniques to handle the replicated system for more complex model architectures, such as the random feature model (Gerace et al, 2020), kernel method (Canatar et al, 2021;Dietrich et al, 1999), and multi-layer neural networks (Schwarze and Hertz, 1992;Yoshino, 2020).…”

Section: Discussionmentioning

confidence: 99%

Sharp Asymptotics of Self-training with Linear Classifier

Takahashi¹

2022

Preprint

View full text Add to dashboard Cite

Self-training (ST) is a straightforward and standard approach in semi-supervised learning, successfully applied to many machine learning problems. The performance of ST strongly depends on the supervised learning method used in the refinement step and the nature of the given data; hence, a general performance guarantee from a concise theory may become loose in a concrete setup. However, the theoretical methods that sharply predict how the performance of ST depends on various details for each learning scenario are limited. This study develops a novel theoretical framework for sharply characterizing the generalization abilities of the models trained by ST using the non-rigorous replica method of statistical physics. We consider the ST of the linear model that minimizes the ridge-regularized cross-entropy loss when the data are generated from a two-component Gaussian mixture. Consequently, we show that the generalization performance of ST in each iteration is sharply characterized by a small finite number of variables, which satisfy a set of deterministic selfconsistent equations. By numerically solving these self-consistent equations, we find that ST's generalization performance approaches to the supervised learning method with a very simple regularization schedule when the label bias is small and a moderately large number of iterations are used.

show abstract

Section: Discussionmentioning

confidence: 99%

Sharp Asymptotics of Self-training with Linear Classifier

Takahashi¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Moreover we expect that dense piecewise linear spheres, at finite temperature, will display strong Gardner phenomenology [18] upon cooling. Finally it would be interesting to see what happens for deeper models beyond the perceptron architecture [19][20][21] as well as more complex constraint satisfaction problems [22].…”

Section: Discussionmentioning

confidence: 99%

Proliferation of non-linear excitations in the piecewise-linear perceptron

Sclocchi

Urbani

2021

SciPost Phys.

View full text Add to dashboard Cite

We investigate the properties of local minima of the energy landscape of a continuous non-convex optimization problem, the spherical perceptron with piecewise linear cost function and show that they are critical, marginally stable and displaying a set of pseudogaps, singularities and non-linear excitations whose properties appear to be in the same universality class of jammed packings of hard spheres. The piecewise linear perceptron problem appears as an evolution of the purely linear perceptron optimization problem that has been recently investigated in [1]. Its cost function contains two non-analytic points where the derivative has a jump. Correspondingly, in the non-convex/glassy phase, these two points give rise to four pseudogaps in the force distribution and this induces four power laws in the gap distribution as well. In addition one can define an extended notion of isostaticity and show that local minima appear again to be isostatic in this phase. We believe that our results generalize naturally to more complex cases with a proliferation of non-linear excitations as the number of non-analytic points in the cost function is increased.

show abstract

“…The phase space volume, which is called Gardner's volume, can be expressed for the present DNN as [1],…”

Section: Gardner's Volumementioning

confidence: 99%

“…For the theoretical part, we use the replica approach developed recently [1] which works on high dimensional data D = N 1. We show that it becomes exact in the dense limit N c 1 and M 1 with fixed α = M/c.…”

Section: Introductionmentioning

confidence: 99%

Spatially heterogeneous learning by a deep student machine

Yoshino¹

2023

Preprint

View full text Add to dashboard Cite

Despite the spectacular successes, deep neural networks (DNN) with a huge number of adjustable parameters remain largely black boxes. To shed light on the hidden layers of DNN, we study supervised learning by a DNN of width N and depth L consisting of perceptrons with c inputs by a statistical mechanics approach called the teacher-student setting. We consider an ensemble of student machines that exactly reproduce M sets of N dimensional input/output relations provided by a teacher machine. We analyze the ensemble theoretically using a replica method ( Hajime Yoshino, SciPost Phys. Core 2, 005 (2020). [1])) and numerically performing greedy Monte Carlo simulations. The replica theory which works on high dimensional data N 1 becomes exact in 'dense limit' N c 1 and M 1 with fixed α = M/c. Both the theory and the simulation suggest learning by the DNN is quite heterogeneous in the network space: configurations of the machines are more correlated within the layers closer to the input/output boundaries while the central region remains much less correlated due to over-parametrization. Deep enough systems relax faster thanks to the less correlated central region. Remarkably both the theory and simulation suggest generalization-ability of the student machines does not vanish even in the deep limit L 1 where the system becomes strongly over-parametrized. We also consider the impact of effective dimension D(≤ N ) of data by incorporating the hidden manifold model (Sebastian Goldt, Marc Mézard, Florent Krzakala, and Lenka Zdevorová, Physical Review X 10, 041044 (2020). [2]) into our model. The replica theory implies that the loop corrections to the dense limit, which reflect correlations between different nodes in the network, become enhanced by either decreasing the width N or decreasing the effective dimension D of the data. Simulation suggests both leads to significant improvements in generalization-ability.

show abstract

From complex to simple : hierarchical free-energy landscape renormalized in deep neural networks

Cited by 9 publications

References 80 publications

Sharp Asymptotics of Self-training with Linear Classifier

Sharp Asymptotics of Self-training with Linear Classifier

Proliferation of non-linear excitations in the piecewise-linear perceptron

Spatially heterogeneous learning by a deep student machine

Contact Info

Product

Resources

About