A class of new three-term descent conjugate gradient algorithms for large-scale unconstrained optimization and applications to image restoration problems

Wang, Xiaoliang; Yuan, Gonglin; Pang, Li-Ping

doi:10.1007/s11075-022-01448-y

Cited by 6 publications

(1 citation statement)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where ∥ • ∥ denote the Euclidean norm and y k+1 = g s+1 − g s . For better theoretical or numerical results, many scholars have revised these classical directions [3,9,10,24,26,27]. Among them, Yuan [26] obtained the global convergence of PRP through a modified wolf line search.…”

Section: Conjugate Gradient Methodsmentioning

confidence: 99%

Stochastic three-term conjugate gradient method with variance technique for non-convex learning

Ouyang,

Lu,

Zhao

et al. 2024

Stat Comput

Self Cite

View full text Add to dashboard Cite

In the training process of machine learning, the minimization of the empirical risk loss function is often used to measure the difference between the model's predicted value and the real value. Stochastic gradient descent is very popular for this type of optimization problem, but converges slowly in theoretical analysis. To solve this problem, there are already many algorithms with variance reduction techniques, such as SVRG, SAG, SAGA, etc. Some scholars apply the conjugate gradient method in traditional optimization to these algorithms, such as CGVR, SCGA, SCGN, etc., which can basically achieve linear convergence speed, but these conclusions often need to be established under some relatively strong assumptions. In traditional optimization, the conjugate gradient method often requires the use of line search techniques to achieve good experimental results. In a sense, line search embodies some properties of the conjugate methods. Taking inspiration from this, we apply the modified three-conjugate gradient method and line search technique to machine learning. In our theoretical analysis, we obtain the same convergence rate as SCGA under weaker conditional assumptions. We also test the convergence of our algorithm using two non-convex machine learning models.

show abstract

Section: Conjugate Gradient Methodsmentioning

confidence: 99%

Stochastic three-term conjugate gradient method with variance technique for non-convex learning

Ouyang,

Lu,

Zhao

et al. 2024

Stat Comput

Self Cite

View full text Add to dashboard Cite

show abstract

An improved Dai‐Liao‐style hybrid conjugate gradient‐based method for solving unconstrained nonconvex optimization and extension to constrained nonlinear monotone equations

Yuan,

Shao,

Zeng

et al. 2024

Math Methods in App Sciences

View full text Add to dashboard Cite

In this work, for unconstrained optimization, we introduce an improved Dai‐Liao‐style hybrid conjugate gradient method based on the hybridization‐based self‐adaptive technique, and the search direction generated fulfills the sufficient descent and trust region properties regardless of any line search. The global convergence is established under standard Wolfe line search and common assumptions. Then, combining the hyperplane projection technique and a new self‐adaptive line search, we extend the proposed conjugate gradient method and obtain an improved Dai‐Liao‐style hybrid conjugate gradient projection method to solve constrained nonlinear monotone equations. Under mild conditions, we obtain its global convergence without Lipschitz continuity. In addition, the convergence rates for the two proposed methods are analyzed, respectively. Finally, numerical experiments are conducted to demonstrate the effectiveness of the proposed methods.

show abstract

A new multi-step method for solving nonlinear systems with high efficiency indices

Erfanifar,

Hajarian

2024

Numer Algor

View full text Add to dashboard Cite

A class of new three-term descent conjugate gradient algorithms for large-scale unconstrained optimization and applications to image restoration problems

Cited by 6 publications

References 37 publications

Stochastic three-term conjugate gradient method with variance technique for non-convex learning

Stochastic three-term conjugate gradient method with variance technique for non-convex learning

An improved Dai‐Liao‐style hybrid conjugate gradient‐based method for solving unconstrained nonconvex optimization and extension to constrained nonlinear monotone equations

A new multi-step method for solving nonlinear systems with high efficiency indices

Contact Info

Product

Resources

About