Effective Dimension Adaptive Sketching Methods for Faster Regularized Least-Squares Optimization

Lacotte, Jonathan; Pilancı, Mert

doi:10.48550/arxiv.2006.05874

Cited by 4 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Notable examples of distributed nonlinear computations that can benefit from Computational Polarization include solving linear systems and convex optimization problems [21], [22], [23], and training neural networks [24]. Another interesting research direction is investigating applications of the Banach space martingales in traditional Polar Codes and improving the existing convergence analysis.…”

Section: Discussionmentioning

confidence: 99%

Computational Polarization: An Information-theoretic Method for Resilient Computing

Pilancı¹

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We introduce an error resilient distributed computing method based on an extension of the channel polarization phenomenon to distributed algorithms. The method leverages an algorithmic split operation that transforms two identical compute nodes to slow and fast workers, which parallels the channel split operation in Polar Codes. This operation preserves the average runtime, analogous to the conservation of Shannon capacity in channel polarization.By leveraging a recursive construction in a similar spirit to the Fast Fourier Transform, this method synthesizes virtual compute nodes with dispersed return time distributions, which we call computational polarization. We show that the runtime distributions form a functional martingale processes, identify their limiting distributions in closedform expressions together with non-asymptotic convergence rates, and prove strong convergence results in Banach spaces. We provide an information-theoretic lower bound on the overall runtime of any coded computation method and show that the computational polarization approach asymptotically achieves the optimal runtime for computing linear functions. An important advantage is the near linear time decoding procedure, which is significantly cheaper than Maximum Distance Separable codes. IndexTerms error correcting codes, Polar codes, coding for computation, random processes, martingales in Banach spaces I. INTRODUCTION As a result of the recent growth of data, the computing paradigm has shifted into massively distributed computing systems. Several distributed architectures and software frameworks have been developed for large scale computational problems. Notable examples include the open source distributed computing framework Apache Spark [1] and

show abstract

Section: Discussionmentioning

confidence: 99%

Computational Polarization: An Information-theoretic Method for Resilient Computing

Pilancı¹

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The SDPs can also be defined and solved in deep learning frameworks with appropriate parameterizations. Random projection and sketching based optimizers for high-dimensional convex programs [39,40,24] and randomized preconditioning [23,27,26,33,28] can address these computational challenges. We leave this as an important open problem.…”

Section: Discussionmentioning

confidence: 99%

Training Quantized Neural Networks to Global Optimality via Semidefinite Programming

Bartan,

Pilanci

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Neural networks (NNs) have been extremely successful across many tasks in machine learning. Quantization of NN weights has become an important topic due to its impact on their energy efficiency, inference time and deployment on hardware. Although post-training quantization is well-studied, training optimal quantized NNs involves combinatorial non-convex optimization problems which appear intractable. In this work, we introduce a convex optimization strategy to train quantized NNs with polynomial activations. Our method leverages hidden convexity in twolayer neural networks from the recent literature, semidefinite lifting, and Grothendieck's identity. Surprisingly, we show that certain quantized NN problems can be solved to global optimality in polynomial-time in all relevant parameters via semidefinite relaxations. We present numerical examples to illustrate the effectiveness of our method. of the SDP. Our techniques lead to a provable guarantee for the difference between the resulting loss and the optimal non-convex loss for the first time. Prior workRecently, there has been a lot of research effort in the realm of compression and quantization of neural networks for real-time implementations. In [41], the authors proposed a method that reduces network weights into ternary values by performing training with ternary values. Experiments in [41] show that their method does not suffer from performance degradation and achieve 16x compression compared to the original model. The authors in [18] focus on compressing dense layers using quantization to tackle model storage problems for large-scale models. The work presented in [21] also aims to compress deep networks using a combination of pruning, quantization and Huffman coding. In [30], the authors present a quantization scheme where they use different bit-widths for different layers (i.e., bit-width optimization). Other works that deal with fixed point training include [29], [19], [22]. Furthermore, [4] proposes layer-wise quantization based on ℓ 2 -norm error minimization followed by retraining of the quantized weights. However, these studies do not address optimal approximation. In comparison, our approach provides optimal quantized neural networks.In [2], it was shown that the degree two polynomial activation functions perform comparably to ReLU activation in practical deep networks. Specifically, it was reported in [2] that for deep neural networks, ReLU activation achieves a classification accuracy of 0.96 and a degree two polynomial activation yields an accuracy of 0.95 on the Cifar-10 dataset. Similarly for the Cifar-100 dataset, it is possible to obtain an accuracy of 0.81 for ReLU activation and 0.76 for the degree two polynomial activation. These numerical results are obtained for the activation σ(t) = t + 0.1t 2 . Furthermore, in encrypted computing, it is desirable to have low degree polynomials as activation functions. For instance, homomorphic encryption methods can only support additions and multiplications in a straightforward way. These constraints mak...

show abstract

“…We review serveral sharp estimates of γ 1 , γ d and discuss the probability that E S holds. For the Gaussian case, we have the following theorem introduced in [14].…”

Section: Sharp Estimates Of Extreme Eigenvalues Of C Smentioning

confidence: 99%

“…Nevertheless, in practice, usually the estimation of d e is available when d e is small, see [16]. Following the adaptive method described in [14], we propose a practical method for finding an appropriate sketching size. We apply IHS to solve (1) with λ = λ min .…”

Section: Estimation Of the Effective Dimensionmentioning

confidence: 99%

Sketching the Krylov Subspace: Faster Computation of the Entire Ridge Regularization Path

Wang

Pilancı

2022

Preprint

Self Cite

View full text Add to dashboard Cite

We propose a fast algorithm for computing the entire ridge regression regularization path in nearly linear time. Our method constructs a basis on which the solution of ridge regression can be computed instantly for any value of the regularization parameter. Consequently, linear models can be tuned via cross-validation or other risk estimation strategies with substantially better efficiency. The algorithm is based on iteratively sketching the Krylov subspace with a binomial decomposition over the regularization path. We provide a convergence analysis with various sketching matrices and show that it improves the state-of-the-art computational complexity. We also provide a technique to adaptively estimate the sketching dimension. This algorithm works for both the over-determined and under-determined problems. We also provide an extension for matrix-valued ridge regression. The numerical results on real medium and large scale ridge regression tasks illustrate the effectiveness of the proposed method compared to standard baselines which require super-linear computational time.

show abstract

Effective Dimension Adaptive Sketching Methods for Faster Regularized Least-Squares Optimization

Cited by 4 publications

References 0 publications

Computational Polarization: An Information-theoretic Method for Resilient Computing

Computational Polarization: An Information-theoretic Method for Resilient Computing

Training Quantized Neural Networks to Global Optimality via Semidefinite Programming

Sketching the Krylov Subspace: Faster Computation of the Entire Ridge Regularization Path

Contact Info

Product

Resources

About