Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations

Abstract

Fast linear transforms are ubiquitous in machine learning, including the discrete Fourier transform, discrete cosine transform, and other structured transformations such as convolutions. All of these transforms can be represented by dense matrix-vector multiplication, yet each has a specialized and highly efficient (subquadratic) algorithm. We ask to what extent hand-crafting these algorithms and implementations is necessary, what structural priors they encode, and how much knowledge is required to automatically learn a fast algorithm for a provided structured transform. Motivated by a characterization of fast matrix-vector multiplication as products of sparse matrices, we introduce a parameterization of divide-and-conquer methods that is capable of representing a large class of transforms. This generic formulation can automatically learn an efficient algorithm for many important transforms; for example, it recovers the O(N log N ) Cooley-Tukey FFT algorithm to machine precision, for dimensions N up to 1024. Furthermore, our method can be incorporated as a lightweight replacement of generic matrices in machine learning pipelines to learn efficient and compressible transformations. On a standard task of compressing a single hidden-layer network, our method exceeds the classification accuracy of unconstrained matrices on CIFAR-10 by 3.9 points-the first time a structured approach has done so-with 4X faster inference speed and 40X fewer parameters.

Butterfly Transform: An Efficient FFT Based Neural Architecture Design

“…The procedure explained in algorithm 1 can be represented by a butterfly graph similar to the FFT's graph. The butterfly network structure has been used for function representation [26] and fast factorization for approximating linear transformation [6]. We adopt this graph as an architecture design for the layers of a neural network.…”

Low-Memory Neural Network Training: A Technical Report

“…Another way to reduce the model (and optimizer) memory required for storing and training a neural network is to replace weight matrices with special structured matrices, such as low-rank matrices [Sainath et al, 2013], Toeplitz-like matrices [Sindhwani et al, 2015], block-circulant matrices [Cheng et al, 2015, Ding et al, 2017, Fastfood transforms [Yang et al, 2015], low displacement rank matrices [Thomas et al, 2018], and butterfly matrices [Dao et al, 2019].…”