Loss Surface Modality of Feed-Forward Neural Network Architectures

Bosman, Anna Sergeevna; Engelbrecht, Andries P.; Helbig, Mardé

doi:10.1109/ijcnn48605.2020.9206727

Cited by 9 publications

(4 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, since distances do not decay fully, this could again imply that trajectories converge to nearby saddle points. We know that saddle points are ubiquitous in high-dimensional systems (both general dynamical systems (Fyodorov and Khoruzhenko, 2016;Ben Arous et al, 2021) and empirically in neural network loss landscapes (Bosman et al, 2020b) and that the convergence of GD slows down near any critical point, whether that is a minima or a saddle. In fact, GD is notoriously bad at escaping saddle points (one of the reasons SGD is preferred in practice).…”

Section: Stability Analysis Near the Stationary State (Post-learning)mentioning

confidence: 99%

Dynamical stability and chaos in artificial neural network trajectories along training

Danovski,

Soriano,

Lacasa

2024

Front. Complex Syst.

View full text Add to dashboard Cite

The process of training an artificial neural network involves iteratively adapting its parameters so as to minimize the error of the network’s prediction, when confronted with a learning task. This iterative change can be naturally interpreted as a trajectory in network space–a time series of networks–and thus the training algorithm (e.g., gradient descent optimization of a suitable loss function) can be interpreted as a dynamical system in graph space. In order to illustrate this interpretation, here we study the dynamical properties of this process by analyzing through this lens the network trajectories of a shallow neural network, and its evolution through learning a simple classification task. We systematically consider different ranges of the learning rate and explore both the dynamical and orbital stability of the resulting network trajectories, finding hints of regular and chaotic behavior depending on the learning rate regime. Our findings are put in contrast to common wisdom on convergence properties of neural networks and dynamical systems theory. This work also contributes to the cross-fertilization of ideas between dynamical systems theory, network theory and machine learning.

show abstract

Section: Stability Analysis Near the Stationary State (Post-learning)mentioning

confidence: 99%

Dynamical stability and chaos in artificial neural network trajectories along training

Danovski,

Soriano,

Lacasa

2024

Front. Complex Syst.

View full text Add to dashboard Cite

show abstract

“…An alternative to random sampling for NN loss landscapes is the progressive gradient walk (PGW) [2,4,5], used in this study. The PGW algorithm is based on the idea of a random walk [23,36].…”

Section: Progressive Gradient Samplingmentioning

confidence: 99%

“…LGCs were successfully used to investigate the minima associated with different loss functions [2] as well as NN architectures [5]. This study is a natural extension of [2,5], where the focus is shifted to the activation functions and their effect on the loss landscapes.…”

Section: Loss-gradient Clouds Loss-gradient Clouds (mentioning

confidence: 99%

Empirical Loss Landscape Analysis of Neural Network Activation Functions

Bosman

Engelbrecht

Helbig

2023

Proceedings of the Companion Conference on Genetic and Evolutionary Computation

Self Cite

View full text Add to dashboard Cite

Activation functions play a significant role in neural network design by enabling non-linearity. The choice of activation function was previously shown to influence the properties of the resulting loss landscape. Understanding the relationship between activation functions and loss landscape properties is important for neural architecture and training algorithm design. This study empirically investigates neural network loss landscapes associated with hyperbolic tangent, rectified linear unit, and exponential linear unit activation functions. Rectified linear unit is shown to yield the most convex loss landscape, and exponential linear unit is shown to yield the least flat loss landscape, and to exhibit superior generalisation performance. The presence of wide and narrow valleys in the loss landscape is established for all activation functions, and the narrow valleys are shown to correlate with saturated neurons and implicitly regularised network configurations. CCS CONCEPTS• Computing methodologies → Neural networks; Continuous space search.

show abstract

“…Bosman et al [16][17][18][19] applied and adapted standard fitness landscape analysis techniques to error landscapes. Studies include: the influence of search space boundaries on the landscape analysis [16], the influence of regularisation on error surfaces [17], the influence of architecture settings on modality of the landscape [18], and the effect of different loss functions on the basins of attraction [19].…”

Section: Error Landscapesmentioning

confidence: 99%

A Survey of Advances in Landscape Analysis for Optimisation

Malan

2021

Algorithms

View full text Add to dashboard Cite

Fitness landscapes were proposed in 1932 as an abstract notion for understanding biological evolution and were later used to explain evolutionary algorithm behaviour. The last ten years has seen the field of fitness landscape analysis develop from a largely theoretical idea in evolutionary computation to a practical tool applied in optimisation in general and more recently in machine learning. With this widened scope, new types of landscapes have emerged such as multiobjective landscapes, violation landscapes, dynamic and coupled landscapes and error landscapes. This survey is a follow-up from a 2013 survey on fitness landscapes and includes an additional 11 landscape analysis techniques. The paper also includes a survey on the applications of landscape analysis for understanding complex problems and explaining algorithm behaviour, as well as algorithm performance prediction and automated algorithm configuration and selection. The extensive use of landscape analysis in a broad range of areas highlights the wide applicability of the techniques and the paper discusses some opportunities for further research in this growing field.

show abstract

Loss Surface Modality of Feed-Forward Neural Network Architectures

Cited by 9 publications

References 16 publications

Dynamical stability and chaos in artificial neural network trajectories along training

Dynamical stability and chaos in artificial neural network trajectories along training

Empirical Loss Landscape Analysis of Neural Network Activation Functions

A Survey of Advances in Landscape Analysis for Optimisation

Contact Info

Product

Resources

About