Periodic step-size adaptation in second-order gradient descent for single-pass on-line structured learning

Abstract

It has been established that the second-order stochastic gradient descent (SGD) method can potentially achieve generalization performance as well as empirical optimum in a single pass through the training examples. However, second-order SGD requires computing the inverse of the Hessian matrix of the loss function, which is prohibitively expensive for structured prediction problems that usually involve a very high dimensional feature space. This paper presents a new second-order SGD method, called Periodic Step-size Adaptation (PSA). PSA approximates the Jacobian matrix of the mapping function and explores a linear relation between the Jacobian and Hessian to approximate the Hessian, which is proved to be simpler and more effective than directly approximating Hessian in an on-line setting. We tested PSA on a wide variety of models and tasks, including large scale sequence labeling tasks using conditional random fields and large scale classification tasks using linear support vector machines and convolutional neural networks. Experimental results show that single-pass performance of PSA is always very close to empirical optimum.

Guest editorial: special issue on structured prediction

“…In many cases, therefore, efficient optimization remains an open problem in structured prediction. Two papers in this special issue (Hsu et al 2009;Sutton and McCallum 2009) address this problem. An important source of complexity in structured prediction algorithms is in the iterative nature of the training step: Often, training is done EM-style, where model parameters are estimated at each step, and then inference is performed based on these parameters.…”

“…Unfortunately, computing the Hessian for loss functions on structured objects involves correlations between labels in different cliques, which is prohibitively expensive in most cases. The paper, "Periodic step-size adaptation in second-order gradient descent for single-pass online structured learning" by Hsu et al (2009) proposes a second-order stochastic gradient descent method in an online setting. The method approximates the Hessian by exploring a linear relation between the Hessian and the Jacobian such that the computation can be performed very efficiently for each online update.…”

Accelerating Generalized Iterative Scaling Based on Staggered Aitken Method for on-Line Conditional Random Fields

“…For converting a conventional batch CRF to an on-line one, several methods have been successfully applied. 8,22,27 In general, the stochastic method used in NNs has been applied to conventional batch CRFs in order to convert them to on-line ones. One type is the Stochastic Gradient Descent (SGD), which has been most successfully applied to CRFs.…”

“…The learning algorithms for batch CRFs are generally based on second-order information requiring computation of the inversion of the Hessian. 8,27 In order to reduce the computation time of the inversion of the Hessian in on-line learning algorithms for CRFs, the Hessian-vector product, 17,27 Quasi-Newton SGD (SGD-QN), 1 Stochastic Meta-Descent (SMD) 27 and Componentwise Triple Jump method for Penalized generalized Iterative Scaling (CTJPIS) 8 were applied.…”

“…Hsu et al 8 proposed an efficient on-and off-line algorithm for CRF learning by approximating the Jacobian of the mapping function's matrix with CTJPIS. Vijayakumar et al 27 applied an SMD with Hessian-vector products to estimate the parameters of a CRF.…”