An experimental evaluation of large scale GBDT systems

Fu, Fang; Jiang, Jiawei; Shao, Yingxia; Cui, Bin

doi:10.14778/3342263.3342273

Cited by 24 publications

(9 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We have also used data parallelism to implement LambdaML. Other research topics in distributed ML include compression [6,7,52,53,93,96,97,101], decentralization [28,41,59,65,90,91,100], synchronization [4,19,26,46,66,68,87,94,102], straggler [8,56,83,89,98,105], data partition [1,3,36,55,77], etc.…”

Section: Related Workmentioning

confidence: 99%

Towards Demystifying Serverless Machine Learning Training

Jiang

Gan

Liu

et al. 2021

Proceedings of the 2021 International Conference on Management of Data

Self Cite

View full text Add to dashboard Cite

The appeal of serverless (FaaS) has triggered a growing interest on how to use it in data-intensive applications such as ETL, query processing, or machine learning (ML). Several systems exist for training large-scale ML models on top of serverless infrastructures (e.g., AWS Lambda) but with inconclusive results in terms of their performance and relative advantage over "serverful" infrastructures (IaaS). In this paper we present a systematic, comparative study of distributed ML training over FaaS and IaaS. We present a design space covering design choices such as optimization algorithms and synchronization protocols, and implement a platform, LambdaML, that enables a fair comparison between FaaS and IaaS. We present experimental results using LambdaML, and further develop an analytic model to capture cost/performance tradeoffs that must be considered when opting for a serverless infrastructure. Our results indicate that ML training pays off in serverless only for models with efficient (i.e., reduced) communication and that quickly converge. In general, FaaS can be much faster but it is never significantly cheaper than IaaS.

show abstract

Section: Related Workmentioning

confidence: 99%

Towards Demystifying Serverless Machine Learning Training

Jiang

Gan

Liu

et al. 2021

Proceedings of the 2021 International Conference on Management of Data

Self Cite

View full text Add to dashboard Cite

show abstract

“…Thun-derGBM also uses the centralized design for a machine with multiple GPUs, but does not support distributed computing. [Fu et al, 2019]. Feature based partitioning does not require the whole histogram to be sent to other machines.…”

Section: Network Communicationmentioning

confidence: 99%

Challenges and Opportunities of Building Fast GBDT Systems

Wen

et al. 2021

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence

Self Cite

View full text Add to dashboard Cite

In the last few years, Gradient Boosting Decision Trees (GBDTs) have been widely used in various applications such as online advertising and spam filtering. However, GBDT training is often a key performance bottleneck for such data science pipelines, especially for training a large number of deep trees on large data sets. Thus, many parallel and distributed GBDT systems have been researched and developed to accelerate the training process. In this survey paper, we review the recent GBDT systems with respect to accelerations with emerging hardware as well as cluster computing, and compare the advantages and disadvantages of the existing implementations. Finally, we present the research opportunities and challenges in designing fast next generation GBDT systems.

show abstract

“…Empirically, 20 buckets are used in popular GBDT frameworks [13,18] The split finding algorithm is depicted in Algorithm 1.…”

Section: Gradient Boosting Decision Tree (Gbdt)mentioning

confidence: 99%

“…• Its efficiency should be close to traditional distributed ML [8,13,18,20], i.e., the number of cryptographic operations should be minimized.…”

Section: Problem Statementmentioning

confidence: 99%

“…The commonest way for distributed bucket construction in traditional distributed GBDT [8,13,18,20] is named quantile sketch [14,19], which requires each participant to send representations of its local data so that the distribution of each feature can be approximated. This approach will inevitably reveal information about participants' local data.…”

Section: Distributed Bucket Constructionmentioning

confidence: 99%

See 1 more Smart Citation

FederBoost: Private Federated Learning for GBDT

Tian¹,

Zhang²,

Hou³

et al. 2020

Preprint

View full text Add to dashboard Cite

An emerging trend in machine learning and artificial intelligence is federated learning (FL), which allows multiple participants to contribute various training data to train a better model. It promises to keep the training data local for each participant, leading to low communication complexity and high privacy. However, there are still two problems in FL remain unsolved: (1) unable to handle vertically partitioned data, and (2) unable to support decision trees. Existing FL solutions for vertically partitioned data or decision trees require heavy cryptographic operations.In this paper, we propose a framework named FederBoost for private federated learning of gradient boosting decision trees (GBDT). It supports running GBDT over both horizontally and vertically partitioned data. The key observation for designing FederBoost is that the whole training process of GBDT relies on the order of the data instead of the values. Consequently, vertical FederBoost does not require any cryptographic operation and horizontal FederBoost only requires lightweight secure aggregation.We fully implement FederBoost and evaluate its utility and efficiency through extensive experiments performed on three public datasets. Our experimental results show that both vertical and horizontal FederBoost achieve the same level of AUC with centralized training where all data are collected in a central server; and both of them can finish training within half an hour even in WAN.

show abstract

An experimental evaluation of large scale GBDT systems

Cited by 24 publications

References 45 publications

Towards Demystifying Serverless Machine Learning Training

Towards Demystifying Serverless Machine Learning Training

Challenges and Opportunities of Building Fast GBDT Systems

FederBoost: Private Federated Learning for GBDT

Contact Info

Product

Resources

About