A hardware acceleration technique for gradient descent and conjugate gradient

Abstract

Gradient descent, conjugate gradient, and other iterative algorithms are a powerful class of algorithms; however, they can take a long time for convergence. Baseline accelerator designs feature insufficient coverage of operations and do not work well on the problems we target. In this thesis we present a novel hardware architecture for accelerating gradient descent and other similar algorithms. To support this architecture, we also present a sparse matrix-vector storage format, and software support for utilizing the format, so that it can be efficiently mapped onto hardware which is also well suited for dense operations. We show that the accelerator design outperforms similar designs which target only the most dominant operation of a given algorithm, providing substantial energy and performance benefits. We further show that the accelerator can be reasonably implemented on a general purpose CPU with small area overhead.ii ACKNOWLEDGMENTS

TABLA: A unified template-based framework for accelerating statistical machine learning

“…Using this insight, TABLA provides a high-level abstraction for programmers to utilize FPGAs as the accelerator of choice for machine learning algorithms without exposing the details of hardware design. There have also been architectures that accelerate gradient descent [47] and conjugate gradient descent [47][48][49][50]. However, these works do not specialize their architectures in machine learning algorithms or any specific objective function.…”

“…Gradient descent accelerators. The work by Kesler [47] focuses only on designing an accelerator suitable for different linear algebra operations to facilitate the gradient descent and conjugate gradient algorithms. Machine learning accelerators.…”

On software design for stochastic processors

“…The quadratic nature of the problem allows for CG to converge in fewer iterations (compared to gradient descent), while also naturally tolerating certain types of errors. Additionally, by using a specialized accelerator for linear operations [9], more applications such as Graph Matching can achieve energy benefits by using stochastic processors. For some problems, such as Sorting, it will be remain difficult however, even with a accelerators, to achieve energy benefits with voltage/scaling type models.…”