Efficient and sparse neural networks by pruning weights in a multiobjective learning approach

Reiners, Malena; Klamroth, Kathrin; Heldmann, Fabian; Stiglmayr, Michael

doi:10.1016/j.cor.2021.105676

Cited by 16 publications

(5 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In a more general setting, multiobjective training approaches were suggested in [20] to trade off between data loss and regularization terms in the context of image recognition. The different characteristics (slope and curvature) of the considered training goals are addressed by enhancing the stochastic multigradient descent approach [17] with pruning strategies, and by combining adap-tive weighted-sum scalarizations with interval bisection.…”

Section: Related Researchmentioning

confidence: 99%

“…Algorithm 1 summarizes the implementation of the bisection enhanced dichotomic search (BEDS) that enhances the dichotomic search scheme by an occasional bisection step. It is based on [20] and considers the fact that NN training with a certain weight parameter α may yield a local minimum, i.e., the training (here the Adam optimizer) may terminate in a local minimum and not in a global minimum as assumed in the general dichotomic search procedure.…”

Section: ] (Own Illustration)mentioning

confidence: 99%

See 1 more Smart Citation

PINN Training using Biobjective Optimization: The Trade-off between Data Loss and Residual Loss

Heldmann¹,

Berkhahn²,

Ehrhardt³

et al. 2023

Preprint

View full text Add to dashboard Cite

Physics informed neural networks (PINNs) have proven to be an efficient tool to represent problems for which measured data are available and for which the dynamics in the data are expected to follow some physical laws.In this paper, we suggest a multiobjective perspective on the training of PINNs by treating the data loss and the residual loss as two individual objective functions in a truly biobjective optimization approach.As a showcase example, we consider COVID-19 predictions in Germany and built an extended susceptibles-infected-recovered (SIR) model with additionally considered leaky-vaccinated and hospitalized populations (SVIHR model) to model the transition rates and to predict future infections. SIR-type models are expressed by systems of ordinary differential equations (ODEs). We investigate the suitability of the generated PINN for COVID-19 predictions and compare the resulting predicted curves with those obtained by applying the method of non-standard finite differences to the system of ODEs and initial data.The approach is applicable to various systems of ODEs that define dynamical regimes. Those regimes do not need to be SIR-type models, and the corresponding underlying data sets do not have to be associated with COVID-19.

show abstract

Section: Related Researchmentioning

confidence: 99%

Section: ] (Own Illustration)mentioning

confidence: 99%

PINN Training using Biobjective Optimization: The Trade-off between Data Loss and Residual Loss

Heldmann¹,

Berkhahn²,

Ehrhardt³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Sparse control methods offer a solution by leveraging the sparsity or locality properties of system dynamics. To address such a problem, Reiners et al studied and verified the efficiency of sparse neural networks (SNNs) [30]. By focusing control inputs on key aspects of the system dynamics, sparse control methods achieve more efficient control.…”

Section: Introductionmentioning

confidence: 99%

A Sparse Neural Network-Based Control Method for Saturated Nonlinear Affine Systems

Zhang,

Yin,

Huo

et al. 2024

Actuators

View full text Add to dashboard Cite

Saturated nonlinear affine systems are widely encountered in many engineering fields. Currently, most control methods on saturated nonlinear affine systems are not specifically designed based on sparsity-based control methodologies, and they might require sparse identification at the beginning stage and applying tracking control afterwards. In this paper, a sparse neural network (SNN)-based control method from an lp-norm (1 ≤ p < 2) optimization perspective is proposed for saturated nonlinear affine systems by taking advantage of the nice properties of primal dual neural networks for optimization. In particular, when p = 1, a new alternative controller based on SNN is derived, encountering computational difficulties distinct from those of another solution set in the basic dual neural network. The convergence properties of such SNN-based controllers are investigated and analyzed to find a control solution. Five illustrative examples further are shown to demonstrate the efficiency of the proposed SNN-based control method for tracking the desired references of saturated nonlinear affine systems. In the practical application scenario involving the UR5 robot control, the trajectory’s average errors are consistently confined to a minimal magnitude of 10−4 m. These findings substantiate the efficacy of the SNN-based control approach proposed for precise tracking control in saturated nonlinear affine systems.

show abstract

“…Due to the centrality of the problem, alternative approaches have been proposed in the literature. Among many, Reiners et al (2022) and Burke and Flanders (1995) explain how a key aspect in the learning and training of deep neural network is the capability of correctly selecting the network architecture. An optimal network structure ensures not only to avoid overparameterization and overfitting, but also to improve the optimization performance.…”

Section: Introductionmentioning

confidence: 99%

“…Burke and Flanders (1995) propose to leverage a new class of networks called ontogenic neural networks that iteratively learn the optimal network structure during backpropagation via prototype units. Conversely, Reiners et al (2022), suggest to consider a biobjective optimization problem. Implementing such a strategy entails differentiating -while training the neural network -between a measure used for prediction accuracy evaluation (e.g., cross-entropy in classification problems), and a penalty function used to assess the total complexity of the network parameters.…”

Section: Introductionmentioning

confidence: 99%

Optimal Deep Neural Networks by Maximization of the Approximation Power

2020

View full text Add to dashboard Cite

Efficient and sparse neural networks by pruning weights in a multiobjective learning approach

Cited by 16 publications

References 13 publications

PINN Training using Biobjective Optimization: The Trade-off between Data Loss and Residual Loss

PINN Training using Biobjective Optimization: The Trade-off between Data Loss and Residual Loss

A Sparse Neural Network-Based Control Method for Saturated Nonlinear Affine Systems

Optimal Deep Neural Networks by Maximization of the Approximation Power

Contact Info

Product

Resources

About