ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure

Abstract

Curvature in form of the Hessian or its generalized Gauss-Newton (GGN) approximation is valuable for algorithms that rely on a local model for the loss to train, compress, or explain deep networks. Existing methods based on implicit multiplication via automatic differentiation or Kronecker-factored block diagonal approximations do not consider noise in the mini-batch. We present VIVIT, a curvature model that leverages the GGN's low-rank structure without further approximations. It allows for efficient computation of eigenvalues, eigenvectors, as well as per-sample first-and second-order directional derivatives. The representation is computed in parallel with gradients in one backward pass and offers a fine-grained cost-accuracy trade-off, which allows it to scale. As examples for VIVIT's usefulness, we investigate the directional gradients and curvatures during training, and how noise information can be used to improve the stability of second-order methods.

Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization

“…The intrinsic low rank structure of the (empirical) Fisher has been exploited in a number of setups by a number of papers including (Agarwal et al, 2019;Goldfarb et al, 2020;Immer et al, 2021;Dangel et al, 2021).…”