The expressivity and training of deep neural networks: Toward the edge of chaos?

Zhang, Gege; Li, Gangwei; Shen, Weining; Zhang, Wensheng

doi:10.1016/j.neucom.2019.12.044

Cited by 8 publications

(4 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the simple attention task, we observed local chaotic dynamics in SM-RC. Chaotic dynamics are of interest in deep-learning research because they increase the expressiveness of learning models [45,46]. Recently, Inoue et al observed transient chaos in transformer models [47].…”

Section: Discussionmentioning

confidence: 99%

Learning Reservoir Dynamics with Temporal Self-Modulation

Sakemi¹,

Yoshimura²,

Matsuki³

et al. 2023

Preprint

View full text Add to dashboard Cite

Reservoir computing (RC) can efficiently process time-series data by transferring the input signal to randomly connected recurrent neural networks (RNNs), which are referred to as a reservoir. The highdimensional representation of time-series data in the reservoir significantly simplifies subsequent learning tasks. Although this simple architecture allows fast learning and facile physical implementation, the learning performance is inferior to that of other state-of-the-art RNN models. In this paper, to improve the learning ability of RC, we propose self-modulated RC (SM-RC), which extends RC by adding a selfmodulation mechanism. The self-modulation mechanism is realized with two gating variables: an input gate and a reservoir gate. The input gate modulates the input signal, and the reservoir gate modulates the dynamical properties of the reservoir. We demonstrated that SM-RC can perform attention tasks where input information is retained or discarded depending on the input signal. We also found that a chaotic state emerged as a result of learning in SM-RC. This indicates that self-modulation mechanisms provide RC with qualitatively different information-processing capabilities. Furthermore, SM-RC outperformed RC in NARMA and Lorentz model tasks. In particular, SM-RC achieved a higher prediction accuracy than RC with a reservoir 10 times larger in the Lorentz model tasks. Because the SM-RC architecture only requires two additional gates, it is physically implementable as RC, providing a new direction for realizing edge AI.

show abstract

Section: Discussionmentioning

confidence: 99%

Learning Reservoir Dynamics with Temporal Self-Modulation

Sakemi¹,

Yoshimura²,

Matsuki³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In the simple attention task, we observed local chaotic dynamics in SM-RC. Chaotic dynamics are of interest in deep-learning research because they increase the expressiveness of learning models [46,47]. Recently, Inoue et al observed transient chaos in transformer models [48].…”

Section: Discussionmentioning

confidence: 99%

Learning Reservoir Dynamics with Temporal Self-Modulation

Sakemi

Nobukawa

Matsuki

et al. 2023

Preprint

View full text Add to dashboard Cite

Reservoir computing (RC) can efficiently process time-series data by transferring the input signal to randomly connected recurrent neural networks (RNNs), which are referred to as a reservoir. The high-dimensional representation of time-series data in the reservoir significantly simplifies subsequent learning tasks. Although this simple architecture allows fast learning and facile physical implementation, the learning performance is inferior to that of other state-of-the-art RNN models. In this study, to improve the learning ability of RC, we propose self-modulated RC (SM-RC) that extends RC by adding a self-modulation mechanism. We demonstrated that SM-RC can perform attention tasks where input information is retained or discarded depending on the input signal. We also found that a chaotic state emerged as a result of learning in SM-RC. Furthermore, SM-RC significantly outperformed RC in NARMA and Lorentz model tasks. Because the SM-RC architecture only requires two additional gates, it is physically implementable as RC, thereby providing a new direction for realizing edge AI.

show abstract

“…The concept of extended criticality was developed further from the concept of edge of chaos [42,147,148], which under the context of biology refers to the phenomenon that the biotic systems are in a diachronic evolving process in regions of the state space that is between chaos and equilibrium. The concept that a DNN computes at edge of chaos is not novel, and existing works [149][150][151][152] have interpreted the training of DNNs as such. Here, although the relationship between circuitsymmetry breaking and edge of chaos has not been discussed before, the discussion is mostly to clarify the criticality in the symmetry-breaking of DNNs.…”

Section: Discussion: Complexity From Adaptive-symmetries Breakingmentioning

confidence: 99%

Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networks

M.¹

2022

Preprint

View full text Add to dashboard Cite

Background: The scientific understanding of complex systems and deep neural networks (DNNs) are among the unsolved important problems of science; and DNNs are evidently complex systems. Meanwhile, conservative symmetry arguably is the most important concept of physics, and P.W. Anderson, Nobel Laureate in physics, speculated that increasingly sophisticated broken symmetry in many-body systems correlates with increasing complexity and functional specialization. Furthermore, in complex systems such as DNA molecules, different nucleotide sequences consist of different weak bonds with similar free energy; and energy fluctuations would break the symmetries that conserve the free energy of the nucleotide sequences, which selected by the environment would lead to organisms with different phenotypes.Purpose: When the molecule is very large, we might speculate that statistically the system poses in a state that would be of equal probability to transit to a large number of adjacent possible states; that is, an adaptive symmetry whose breaking is selected by the feedback signals from the environment. In physics, quantitative changes would accumulate into qualitative revolution where previous paradoxical behaviors are reconciled under a new paradigm of higher dimensionality (e.g., wave-particle duality in quantum physics). This emergence of adaptive symmetry and complexity might be speculated as accumulation of sophistication and quantity of conservative symmetries that lead to a paradigm shift, which might clarify the behaviors of DNNs.Results: In this work, theoretically and experimentally, we characterize the optimization process of a DNN system as an extended symmetry-breaking process where novel examples are informational perturbations to the system that breaks adaptive symmetries. One particular finding is that a hierarchically large DNN would have a large reservoir of adaptive symmetries, and when the information capacity of the reservoir exceeds the complexity of the dataset, the system could absorb all perturbations of the examples and self-organize into a functional structure of zero training errors measured by a certain surrogate risk. In this diachronically extended process, complexity emerges from quantitative accumulation of adaptive-symmetries breaking.Method: More specifically, this process is characterized by a statistical-mechanical model that could be appreciated as a generalization of statistics physics to the DNN organized complex system, and characterizes regularities in higher dimensionality. The model consists of three constitutes that could be appreciated as the counterparts of Boltzmann distribution, Ising model, and conservative symmetry, respectively: (1) a stochastic definition/interpretation of DNNs that is a multilayer probabilistic graphical model, (2) a formalism of circuits that perform biological computation, (3) a circuit symmetry from which self-similarity between the microscopic and the macroscopic adaptability manifests. The model is analyzed with a method referred as the statistical as...

show abstract

The expressivity and training of deep neural networks: Toward the edge of chaos?

Cited by 8 publications

References 36 publications

Learning Reservoir Dynamics with Temporal Self-Modulation

Learning Reservoir Dynamics with Temporal Self-Modulation

Learning Reservoir Dynamics with Temporal Self-Modulation

Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networks

Contact Info

Product

Resources

About