Fast and Accurate Least-Mean-Squares Solvers for High Dimensional Data

Maalouf, Alaa; Jubran, Ibrahim; Feldman, Danny

doi:10.1109/tpami.2021.3139612

Cited by 26 publications

(52 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…learner encoder url kdda kddb kdd12 For small d, this opens up sketching approaches to learning whose superlinearity in input dimension otherwise makes them inaccessible to wide, sparse datasets, such as coreset construction for nearest-neighbor queries [28]. Further, in the case of linear models, an n × d design can be represented faithfully as a (d + 1) × d one by Carathéodory's Theorem [29], which for small d can greatly simplify linear learning (e.g., tuning regularization parameters no longer requires multiple passes over the data).…”

Section: Discussionmentioning

confidence: 99%

Chromatic Learning for Sparse Datasets

Feinberg,

Bailis

2020

Preprint

View full text Add to dashboard Cite

Learning over sparse, high-dimensional data frequently necessitates the use of specialized methods such as the hashing trick. In this work, we design a highly scalable alternative approach that leverages the low degree of feature co-occurrences present in many practical settings. This approach, which we call Chromatic Learning (CL), obtains a low-dimensional dense feature representation by performing graph coloring over the co-occurrence graph of features-an approach previously used as a runtime performance optimization for GBDT training [1]. This colorbased dense representation can be combined with additional dense categorical encoding approaches, e.g., submodular feature compression, to further reduce dimensionality [2]. CL exhibits linear parallelizability and consumes memory linear in the size of the co-occurrence graph. By leveraging the structural properties of the co-occurrence graph, CL can compress sparse datasets, such as KDD Cup 2012, that contain over 50M features down to 1024, using an order of magnitude fewer features than frequency-based truncation and the hashing trick while maintaining the same test error for linear models. This compression further enables the use of deep networks in this wide, sparse setting, where CL similarly has favorable performance compared to existing baselines for budgeted input dimension.Preprint. Under review.

show abstract

Section: Discussionmentioning

confidence: 99%

Chromatic Learning for Sparse Datasets

Feinberg,

Bailis

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Other related work considers generalizations of k-median and k-means by either adding capacity constraints [7,29,52,85], generalizing the notion of centers to subspaces [19,41,42], time series [54] or sets [61] or considering more general objective functions [5,15]. Coresets have also been studied for many other problems: we cite non-comprehensively decision trees [60], kernel methods [59,62,83], determinant maximization [57], diversity maximization [58], shape fitting problems [2,22], linear regression [9,53,88], logistic regression [56,81], Gaussian mixtures [70], dependency networks [79], or low-rank approximation [71]. The interested reader is referred to [3,40,80] and similar surveys for more pointers to coreset literature.…”

Section: Overview Of Our Techniquesmentioning

confidence: 99%

Towards Optimal Lower Bounds for k-median and k-means Coresets

Cohen-Addad¹,

Larsen²,

Saulpic³

et al. 2022

Preprint

View full text Add to dashboard Cite

Given a set of points in a metric space, the (k, z)-clustering problem consists of finding a set of k points called centers, such that the sum of distances raised to the power of z of every data point to its closest center is minimized. Special cases include the famous k-median problem (z = 1) and k-means problem (z = 2). The k-median and k-means problems are at the heart of modern data analysis and massive data applications have given raise to the notion of coreset: a small (weighted) subset of the input point set preserving the cost of any solution to the problem up to a multiplicative (1 ± ε) factor, hence reducing from large to small scale the input to the problem.While there has been an intensive effort to understand what is the best coreset size possible for both problems in various metric spaces, there is still a significant gap between the stateof-the-art upper and lower bounds. In this paper, we make progress on both upper and lower bounds, obtaining tight bounds for several cases, namely:• In finite n point general metrics, any coreset must consist of Ω(k log n/ε 2 ) points. This improves on the Ω(k log n/ε) lower bound of Braverman, Jiang, Krauthgamer, and Wu [ICML'19] and matches the upper bounds proposed for k-median by Feldman and Langberg [STOC'11] and k-means by Cohen-Addad, Saulpic, and Schwiegelshohn [STOC'21] up to polylog factors. • For doubling metrics with doubling constant D, any coreset must consist of Ω(kD/ε 2 ) points. This matches the k-median and k-means upper bounds by Cohen-Addad, Saulpic, and Schwiegelshohn [STOC'21] up to polylog factors. • In d-dimensional Euclidean space, any coreset for (k, z) clustering requires Ω(k/ε 2 ) points.This improves on the Ω(k/ √ ε) lower bound of Baker, Braverman, Huang, Jiang, Krauthgamer, and Wu [ICML'20] for k-median and complements the Ω(k min(d, 2 z/20 )) lower bound of Huang and Vishnoi [STOC'20]. We complement our lower bound for d-dimensional Euclidean space with the construction of a coreset of size Õ(k/ε 2 • min(ε −z , k)). This improves over the Õ(k 2 ε −4 ) upper bound for general power of z proposed by Braverman Jiang, Krauthgamer, and Wu [SODA'21] and over the Õ(k/ε 4 ) upper bound for k-median by Huang and Vishnoi [STOC'20]. In fact, ours is the first construction breaking through the ε −2 • min(d, ε −2 ) barrier inherent in all previous coreset constructions. To do this, we employ a novel chaining based analysis that may be of independent interest. Together our upper and lower bounds for k-median in Euclidean spaces are tight up to a factor O(ε −1 polylog k/ ).

show abstract

“…In the past years, coreset techniques have been widely applied to many optimization problems, such as: clustering [11,22,25], logistic regression [26,37,42,47,42], Bayesian methods [9,8], linear regression [17,19,12,27,47], robust optimization [18], Gaussian mixture model [32], and active learning [14,43]. Recently, [33] also proposed the notion of "accurate" coresets, which do not introduce any approximation error when compressing the input dataset. Coresets are also applied to speed up large-scale or distributed machine learning algorithms [41,34,35,6].…”

Section: Related Workmentioning

confidence: 99%

A Novel Sequential Coreset Method for Gradient Descent Algorithms

Huang¹,

Huang²,

liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

A wide range of optimization problems arising in machine learning can be solved by gradient descent algorithms, and a central question in this area is how to efficiently compress a large-scale dataset so as to reduce the computational complexity. Coreset is a popular data compression technique that has been extensively studied before. However, most of existing coreset methods are problem-dependent and cannot be used as a general tool for a broader range of applications. A key obstacle is that they often rely on the pseudo-dimension and total sensitivity bound that can be very high or hard to obtain. In this paper, based on the "locality" property of gradient descent algorithms, we propose a new framework, termed "sequential coreset", which effectively avoids these obstacles. Moreover, our method is particularly suitable for sparse optimization whence the coreset size can be further reduced to be only poly-logarithmically dependent on the dimension. In practice, the experimental results suggest that our method can save a large amount of running time compared with the baseline algorithms.

show abstract

Fast and Accurate Least-Mean-Squares Solvers for High Dimensional Data

Cited by 26 publications

References 23 publications

Chromatic Learning for Sparse Datasets

Chromatic Learning for Sparse Datasets

Towards Optimal Lower Bounds for k-median and k-means Coresets

A Novel Sequential Coreset Method for Gradient Descent Algorithms

Contact Info

Product

Resources

About