Gradient Properties of Hard Thresholding Operator

Abstract

Sparse optimization receives increasing attention in many applications such as compressed sensing, variable selection in regression problems, and recently neural network compression in machine learning. For example, the problem of compressing a neural network is a bi-level, stochastic, and nonconvex problem that can be cast into a sparse optimization problem. Hence, developing efficient methods for sparse optimization plays a critical role in applications. The goal of this paper is to develop analytical techniques for general, large size sparse optimization problems using the hard thresholding operator. To this end, we study the iterative hard thresholding (IHT) algorithm, which has been extensively studied in the literature because it is scalable, fast, and easily implementable. In spite of extensive research on the IHT scheme, we develop several new techniques that not only recover many known results but also lead to new results. Specifically, we first establish a new and critical gradient descent property of the hard thresholding (HT) operator. Our gradient descent result can be related to the distance between points that are sparse. However, the distance between sparse points cannot provide any information about the gradient in the sparse setting. To the best of our knowledge, the other way around (the gradient to the distance) has not been shown so far in the literature. Also, our gradient descent property allows one to study the IHT when the stepsize is less than or equal to 1/L, where L>0 is the Lipschitz constant of the gradient of an objective function. Note that the existing techniques in the literature can only handle the case when the stepsize is strictly less than 1/L. By exploiting this we introduce and study HT-stable and HT-unstable stationary points and show no matter how close an initialization is to a HT-unstable stationary point (saddle point in sparse sense), the IHT sequence leaves it. Finally, we show that no matter what sparse initial point is selected, the IHT sequence converges if the function values at HT-stable stationary points are distinct, where the last condition is a new assumption that has not been found in the literature. We provide a video of 4000 independent runs where the IHT algorithm is initialized very close to a HT-unstable stationary point and show the sequences escape them.

The Backpropagation algorithm for a math student

“…These results can aid research on compressing DNNs that utilize the full gradient, as noted in Damadi et al (2022). It can benefit sparse optimization in both deterministic and stochastic settings, as the Iterative Hard Thresholding (IHT) algorithm uses the full gradient for a sparse solution Damadi & Shen (2022b) in deterministic settings and the mini-batch Stochastic IHT algorithm is employed in the stochastic context Damadi & Shen (2022a). We provided concise mathematical justifications to make the results clear and useful for people from different fields, even if they did not have a deep understanding of the mathematics involved.…”

“…For the special case of the ReLU function J z f (z) is a diagonal matrix of zeros and ones which are associated to the negative and positive elements of z. Multiplying such a matrix from the left to any matrix W results in removing the rows of W associated to zero elements in J z f (z) which greatly decreases the computation. The zero-th norm of the parameter vector can be minimized in sparse optimizations using these intuitions, as noted in Damadi & Shen (2022b).…”

RevUp: Revise and Update Information Bottleneck for Event Representation

“…By setting a low temperature of 0.5, the generated Gumbel-Softmax samples become almost identical to one-hot vectors, eliminating the need for gradient approximation. The effects on the gradient were carefully analyzed and the results are presented in Damadi and Shen (2022), where a comprehensive study of gradient properties was conducted. To learn richer representations, we define an embedding matrix E ∈ R T ×dt , to convert a simplex frame into a vector representation as e m = t T m E.…”