Insights into machine learning models from chemical physics: an energy landscapes approach (EL for ML)

Abstract

In this work, we outline how methods from the energy landscapes field of theoretical chemistry can be applied to study machine learning models. Various applications are found, ranging from interpretability to improved model performance.

Explainable Gaussian processes: a loss landscape perspective

“…Insights into the structure and geometric properties of the loss landscape have been employed to improve model performance [13,16], adversarial robustness [17], interpretability [18,19], and generalisability [20][21][22] of neural networks. An overview focusing on the role of loss landscapes in machine learning, and the link to the physical sciences, is given in [15]. Empirically, loss landscape characteristics have also been considered to explain fundamental aspects of machine learning, such as the quality of minima [23] and the structure of the solution space as a function of hyperparameters [12].…”

“…We discuss a novel ensembling approach that aims to tackle both issues. The ensemble scheme considered here is based on the loss landscapes described in previous sections, and a detailed account of the ideas behind landscape-inspired ensembles in given in [15]. Usually, the multiple ML models that form an ensemble result from repeated stochastic gradient descent beginning from new random initialisations of the model.…”

“…Here, we study loss landscapes using tools from the physical sciences [11], where molecular energy landscapes are employed to predict thermodynamic and kinetic properties of a system. We will treat ML loss landscapes analogously, as in previous work [12][13][14], which identified various similarities between molecular energy landscapes and machine learning loss landscapes; a tutorial review on these applications is available in [15].…”

A physics-inspired approach to the understanding of molecular representations and models

“…20 Recently, this methodology has been extended by some of the authors to selected tasks in machine learning such as clustering, 21 and hyperparameter tuning in Gaussian processes, 22 for which we point interested readers to a recent tutorial review. 23 In this contribution, we develop a novel roughness measure inspired by the similarities between model response surfaces and energy landscapes. We describe a method for representing discrete molecular datasets as a continuous surface and encoding the surface topography as a weighted graph.…”