Energy landscapes for machine learning

Ballard, Andrew J.; Das, Ritankar; Martiniani, Stefano; Mehta, Dhagash; Sagun, Levent; Stevenson, Jacob D.; Wales, David J.

doi:10.1039/c7cp01108c

Cited by 95 publications

(107 citation statements)

References 125 publications

Supporting

Mentioning

105

Contrasting

Order By: Relevance

“…Ref. [36] provides a review, highlighting the connections between molecular potential energy surfaces and ANN landscapes.…”

Section: Introductionmentioning

confidence: 99%

Loss surface of XOR artificial neural networks

Mehta¹,

Zhao²,

Bernal³

et al. 2018

Phys. Rev. E

Self Cite

View full text Add to dashboard Cite

Training an artificial neural network involves an optimization process over the landscape defined by the cost (loss) as a function of the network parameters. We explore these landscapes using optimization tools developed for potential energy landscapes in molecular science. The number of local minima and transition states (saddle points of index one), as well as the ratio of transition states to minima, grow rapidly with the number of nodes in the network. There is also a strong dependence on the regularization parameter, with the landscape becoming more convex (fewer minima) as the regularization term increases. We demonstrate that in our formulation, stationary points for networks with N_{h} hidden nodes, including the minimal network required to fit the XOR data, are also stationary points for networks with N_{h}+1 hidden nodes when all the weights involving the additional node are zero. Hence, smaller networks trained on XOR data are embedded in the landscapes of larger networks. Our results clarify certain aspects of the classification and sensitivity (to perturbations in the input data) of minima and saddle points for this system, and may provide insight into dropout and network compression.

show abstract

“…Ref. [36] provides a review, highlighting the connections between molecular potential energy surfaces and ANN landscapes.…”

Section: Introductionmentioning

confidence: 99%

Loss surface of XOR artificial neural networks

Mehta¹,

Zhao²,

Bernal³

et al. 2018

Phys. Rev. E

Self Cite

View full text Add to dashboard Cite

show abstract

“…Well-known examples include a myriad of biophysical processes 4-11 , multiphase systems 2 , thermally activated hopping in optical traps 12,13 , chemical reactions 1,14 , brain neuronal expression 15 , cellular development [16][17][18][19][20] and social networks 21,22 . Energetic concepts have also been connected to machine learning 23 and to viral fitness landscapes, where pathways with the lowest energy barriers may explain typical mutational evolutionary trajectories of viruses between fitness peaks 24,25 . Recent advances in experimental techniques including cryo-electron microscopy (cryo-EM) 4, [26][27][28][29][30] and single-cell RNA sequencing 31 , as well as new online social interaction datasets 32,33 , are producing an unprecedented wealth of high-dimensional instantaneous snapshots of biophysical and social systems.…”

Section: Introductionmentioning

confidence: 99%

Learning dynamical information from static protein and sequencing data

Pearce

Woodhouse

Forrow

et al. 2018

Preprint

View full text Add to dashboard Cite

Many complex processes, from protein folding and virus evolution to brain activity and neuronal network dynamics, can be described as stochastic exploration of a high-dimensional energy landscape. While efficient algorithms for cluster detection and data completion in high-dimensional spaces have been developed and applied over the last two decades, considerably less is known about the reliable inference of state transition dynamics in such settings. Here, we introduce a flexible and robust numerical framework to infer Markovian transition networks directly from time-independent data sampled from stationary equilibrium distributions. Our approach combines Gaussian mixture approximations and self-consistent dimensionality reduction with minimal-energy path estimation and multi-dimensional transition-state theory. We demonstrate the practical potential of the inference scheme by reconstructing the network dynamics for several protein folding transitions, gene regulatory network motifs and HIV evolution pathways. The predicted network topologies and relative transition time scales agree well with direct estimates from time-dependent molecular dynamics data, stochastic simulations and phylogenetic trees, respectively. The underlying numerical protocol thus allows the recovery of relevant dynamical information from instantaneous ensemble measurements, effectively alleviating the need for time-dependent data in many situations. Owing to its generic structure, the framework introduced here will be applicable to high-throughput RNA and protein sequencing datasets and future cryo-electron-microscopy data, and can guide the design of new experimental approaches towards studying complex multiphase phenomena.

show abstract

“…A related idea also appear in the deep learning and statistical physics, which is searching for patterns in some statistical models energy landscape (for instance, see [41][42][43]). There problem is related to the solution of a minimal (local or global) in a high dimensional surface (where is called landscape), or specifically finding a minimal energy in some models from statistical physics, and the high dimension comes from the large N setup in the corresponding model.…”

Section: Jhep12(2017)149mentioning

confidence: 99%

“…As one of the most promising area in computer science, machine learning (or artificial intelligence) [29][30][31] is widely used to understand difference areas of physics, including condensed matter phases [39], high energy experiment [40], energy landscape [41][42][43], particle phenomenology [44], tensor networks [45] and cosmic non-Gaussianities [46]. In recent research string theorists also find that machine learning algorithm is efficient to study manifold data in the string landscape [47][48][49][50], which may give us the motivation to think about the learning algorithm landscape from cosmological point of view.…”

Section: Jhep12(2017)149mentioning

confidence: 99%

Artificial neural network in cosmic landscape

Liu

2017

J. High Energ. Phys.

View full text Add to dashboard Cite

In this paper we propose that artificial neural network, the basis of machine learning, is useful to generate the inflationary landscape from a cosmological point of view. Traditional numerical simulations of a global cosmic landscape typically need an exponential complexity when the number of fields is large. However, a basic application of artificial neural network could solve the problem based on the universal approximation theorem of the multilayer perceptron. A toy model in inflation with multiple light fields is investigated numerically as an example of such an application.

show abstract

Energy landscapes for machine learning

Cited by 95 publications

References 125 publications

Loss surface of XOR artificial neural networks

Loss surface of XOR artificial neural networks

Learning dynamical information from static protein and sequencing data

Artificial neural network in cosmic landscape

Contact Info

Product

Resources

About