Adaptive cubic regularization methods with dynamic inexact Hessian information and applications to finite-sum minimization

Abstract

We consider the adaptive regularization with cubics approach for solving nonconvex optimization problems and propose a new variant based on inexact Hessian information chosen dynamically. The theoretical analysis of the proposed procedure is given. The key property of ARC framework, constituted by optimal worst-case function/derivative evaluation bounds for first- and second-order critical point, is guaranteed. Application to large-scale finite-sum minimization based on subsampled Hessian is discussed and analyzed in both a deterministic and probabilistic manner, and equipped with numerical experiments on synthetic and real datasets.

A stochastic first-order trust-region method with inexact restoration for finite-sum minimization

“…Results are reported in Table 4. 4. We can see that, reducing N 0 , the number of full function/gradient evaluations reduces as well, and that for N 0 = 0.01N the average classification error compares well with the error when N 0 = 0.1N ; for instance, the best results for cina0 and covertype are obtained by shrinking N 0 to 1% of the maximum sample size.…”

“…They do not call for function evaluations but require tuning the learning rate and further possible hyper-parameters such as the mini-batch size. Since the tuning effort may be very computationally demanding [15], more sophisticated approaches use linesearch or trust-region strategies to adaptively choose the learning rate and to avoid tuning efforts, see [2,4,5,9,14,15,25]. In this context, function and gradient approximations have to satisfy sufficient accuracy requirements with some probability.…”

“…The issue of choosing the sample size such that (2.2) and (2.3) hold in probability is delicate. Dynamic strategies for choosing the sample size have been proposed, see e.g., [4,12,14,25]. Considering, for sake of brevity, the estimate…”

“…Applying the above estimate for the sample size is not trivial since in general the upper bound V f on the variance is not available; however it can be replaced with variance estimates [12] obtained during the computation of the subsampled function values, or with estimated upper bounds on |f N (x k )| and ∇f N (x k ) [4,29]. Alternative proposals consist in increasing the sample size by a prefixed factor, or geometrically with the iteration index k by a rule of the form a k for some a > 1, [11,12].…”

The Impact of Noise on Evaluation Complexity: The Deterministic Trust-Region Case

“…This paper attempts to answer a simple question: how does noise in function values and derivatives affect evaluation complexity of smooth optimization? While analysis has been produced to indicate how high accuracy can be reached by optimization algorithms even in the presence of inexact but deterministic (1) function and derivatives' values (see [8,16,28,3,29,21,14]), these approaches crucially rely on the assumption that the inexactness is controllable, in that it can be made arbitrarily small if required so by the algorithm. But what happens in practical applications where significant noise is intrinsic and can't be assumed away?…”