Compressed linear algebra for large-scale machine learning

Elgohary, Ahmed; Böehm, Matthias; Haas, Peter J.; Reiss, Frederick; Reinwald, Berthold

doi:10.14778/2994509.2994515

Cited by 55 publications

(27 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unlike existing work [9,22,48], we made the conscious design decision not to generate the data access into the fused operators. Instead, the handcoded skeleton implements the data access-depending on its sparse-safeness over cells or non-zero values-of dense, sparse, or compressed [28] matrices and calls an abstract (virtual) genexec method for each value. Generated operators inherit this skeleton and only override the specific genexec, which yields very lean yet efficient operators.…”

Section: Code Generation Plansmentioning

confidence: 99%

See 1 more Smart Citation

On optimizing operator fusion plans for large-scale machine learning in systemML

et al. 2018

Self Cite

View full text Add to dashboard Cite

Many large-scale machine learning (ML) systems allow specifying custom ML algorithms by means of linear algebra programs, and then automatically generate efficient execution plans. In this context, optimization opportunities for fused operators-in terms of fused chains of basic operators-are ubiquitous. These opportunities include (1) fewer materialized intermediates, (2) fewer scans of input data, and (3) the exploitation of sparsity across chains of operators. Automatic operator fusion eliminates the need for hand-written fused operators and significantly improves performance for complex or previously unseen chains of operations. However, existing fusion heuristics struggle to find good fusion plans for complex DAGs or hybrid plans of local and distributed operations. In this paper, we introduce an optimization framework for systematically reason about fusion plans that considers materialization points in DAGs, sparsity exploitation, different fusion template types, as well as local and distributed operations. In detail, we contribute algorithms for (1) candidate exploration of valid fusion plans, (2) cost-based candidate selection, and (3) code generation of local and distributed operations over dense, sparse, and compressed data. Our experiments in SystemML show end-toend performance improvements with optimized fusion plans of up to 21x compared to hand-written fused operators, with negligible optimization and code generation overhead.

show abstract

Section: Code Generation Plansmentioning

confidence: 99%

“…Compressed Linear Algebra (CLA): All templates support operations over compressed matrices (column-wise compression, heterogeneous encoding formats, and column co-coding) [28]. Figure 9 shows the runtime of Base, Fused, and Gen for computing the sparse-safe expression sum(X 2 ) over Airline78 and Mnist8m.…”

Section: Operations Performancementioning

confidence: 99%

On optimizing operator fusion plans for large-scale machine learning in systemML

et al. 2018

Self Cite

View full text Add to dashboard Cite

show abstract

“…ShinyLearner is limited to datasets that fit into computer memory. For larger datasets, frameworks such as Apache SystemML support distributed algorithm execution[79]; however, the number of algorithms implemented in these frameworks is still relatively small.…”

Section: Discussionmentioning

confidence: 99%

ShinyLearner: A containerized benchmarking tool for machine-learning classification of tabular data

Piccolo

Lee

Suh

et al. 2019

Preprint

View full text Add to dashboard Cite

AbstractClassification algorithms assign observations to groups based on patterns in data. The machine-learning community have developed myriad classification algorithms, which are employed in diverse life-science research domains. When applying such algorithms, researchers face the challenge of deciding which algorithm(s) to apply in a given research domain. Algorithm choice can affect classification accuracy dramatically, so it is crucial that researchers optimize these choices based on empirical evidence rather than hearsay or anecdotal experience. In benchmark studies, multiple algorithms are applied to multiple datasets, and the researcher examines overall trends. In addition, the researcher may evaluate multiple hyperparameter combinations for each algorithm and use feature selection to reduce data dimensionality. Although software implementations of classification algorithms are widely available, robust benchmark comparisons are difficult to perform when researchers wish to compare algorithms that span multiple software packages.Programming interfaces, data formats, and evaluation procedures differ across software packages; and dependency conflicts may arise during installation. To address these challenges, we created ShinyLearner, an open-source project for integrating machine-learning packages into software containers. ShinyLearner provides a uniform interface for performing classification, irrespective of the library that implements each algorithm, thus facilitating benchmark comparisons. In addition, ShinyLearner enables researchers to optimize hyperparameters and select features via nested cross validation; it tracks all nested operations and generates output files that make these steps transparent. ShinyLearner includes a Web interface to help users more easily construct the commands necessary to perform benchmark comparisons. ShinyLearner is freely available at https://github.com/srp33/ShinyLearner.

show abstract

“…These ideas have been applied to speed up ML workloads. SystemML [24,40,48] ScalOps [98], Pig latin [79], and KeystoneML [92] propose high-level ML languages for automatic parallelization and materialization, as well as easier programming. Hamlet [64] and others [63,86] avoid expensive denormalizations.…”

Section: Dbms-inspired Optimizationmentioning

confidence: 99%

BlinkML

Park

Qing

Shen

et al. 2019

Proceedings of the 2019 International Conference on Management of Data

View full text Add to dashboard Cite

The rising volume of datasets has made training machine learning (ML) models a major computational cost in the enterprise. Given the iterative nature of model and parameter tuning, many analysts use a small sample of their entire data during their initial stage of analysis to make quick decisions (e.g., what features or hyperparameters to use) and use the entire dataset only in later stages (i.e., when they have converged to a specific model). This sampling, however, is performed in an ad-hoc fashion. Most practitioners cannot precisely capture the effect of sampling on the quality of their model, and eventually on their decision-making process during the tuning phase. Moreover, without systematic support for sampling operators, many optimizations and reuse opportunities are lost.In this paper, we introduce BlinkML, a system for fast, quality-guaranteed ML training. BlinkML allows users to make error-computation tradeoffs: instead of training a model on their full data (i.e., full model), BlinkML can quickly train an approximate model with quality guarantees using a sample. The quality guarantees ensure that, with high probability, the approximate model makes the same predictions as the full model. BlinkML currently supports any ML model that relies on maximum likelihood estimation (MLE), which includes Generalized Linear Models (e.g., linear regression, logistic regression, max entropy classifier, Poisson regression) as well as PPCA (Probabilistic Principal Component Analysis). Our experiments show that BlinkML can speed up the training of large-scale ML tasks by 6.26×-629× while guaranteeing the same predictions, with 95% probability, as the full model.

show abstract

Compressed linear algebra for large-scale machine learning

Cited by 55 publications

References 43 publications

On optimizing operator fusion plans for large-scale machine learning in systemML

On optimizing operator fusion plans for large-scale machine learning in systemML

ShinyLearner: A containerized benchmarking tool for machine-learning classification of tabular data

BlinkML

Contact Info

Product

Resources

About