State space representation and phase analysis of gradient descent optimizers

Yao, Biyuan; Li, Guiqing; Wu, Wei

doi:10.1007/s11432-022-3539-8

Sci. China Inf. Sci.

2023

DOI: 10.1007/s11432-022-3539-8

|View full text |Cite

State space representation and phase analysis of gradient descent optimizers

Biyuan Yao

Guiqing Li

Wei Wu

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2024

Publication Types

Select...

Article3

Relationship

Self Cite0

Independent3

Authors

Journals

Cited by 3 publications

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

System response curve based first‐order optimization algorithms for cyber‐physical‐social intelligence

Yao,

Zhang,

Feng

et al. 2024

Concurrency and Computation

View full text Add to dashboard Cite

The continuous enhancement of optimization algorithms and their parameters has spurred the expansion of AI into novel application domains such as image recognition and smart home technology. This paper employs the system response curve (SRC) to the adaptive learning rate optimizer, addressing challenges associated with the establishment of the optimizer control model and parameter adjustments affecting the dynamic performance of the system. These insights offer theoretical support for the optimizer's application in deep learning models. To begin, the adaptive learning rate optimizer is a time‐varying system. Based on the intrinsic relationship between the network optimization and the control system, the time domain expression and approximate transfer function of the adaptive learning rate optimizer are derived, and the system dynamic model is established. Furthermore, based on the system control model of the optimizer, it is proposed to explain the performance impacts of different optimizers and their hyperparameters on the deep learning model through the SRC. Finally, experiments are performed on the MNIST, CIFAR‐10, UTKinect‐Action3D, and Florence3D‐Action datasets to validate the control theory of explaining optimizers through system response curves. The experimental results show that the recognition performance of the Adaptive Moment Estimate (Adam) is better than that of the Adaptive Gradient (AdaGrad) and Root Mean Square Propagation (RMSprop). Additionally, the learning rate affects the model training speed, and the practical application aligns with the theoretical analysis.

show abstract

System response curve based first‐order optimization algorithms for cyber‐physical‐social intelligence

Yao,

Zhang,

Feng

et al. 2024

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

Adaptive PI Control Based Stability Margin Configuration of Aircraft Control Systems with Unknown System Parameters and Time Delay

Zhang,

Zhou,

Wang

et al. 2024

J Syst Sci Complex

View full text Add to dashboard Cite

Design of smart home system speech emotion recognition model based on ensemble deep learning and feature fusion

Wang,

Ma,

Wang

et al. 2024

Applied Acoustics

View full text Add to dashboard Cite

scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.

Contact Info

customersupport@researchsolutions.com

10624 S. Eastern Ave., Ste. A-614

Henderson, NV 89052, USA

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Blog Terms and Conditions API Terms Privacy Policy Contact Cookie Preferences Do Not Sell or Share My Personal Information

Made with 💙 for researchers

Part of the Research Solutions Family.

State space representation and phase analysis of gradient descent optimizers

Cited by 3 publications

References 25 publications

System response curve based first‐order optimization algorithms for cyber‐physical‐social intelligence

System response curve based first‐order optimization algorithms for cyber‐physical‐social intelligence

Adaptive PI Control Based Stability Margin Configuration of Aircraft Control Systems with Unknown System Parameters and Time Delay

Design of smart home system speech emotion recognition model based on ensemble deep learning and feature fusion

Contact Info

Product

Resources

About