FederBoost: Private Federated Learning for GBDT

Abstract

An emerging trend in machine learning and artificial intelligence is federated learning (FL), which allows multiple participants to contribute various training data to train a better model. It promises to keep the training data local for each participant, leading to low communication complexity and high privacy. However, there are still two problems in FL remain unsolved: (1) unable to handle vertically partitioned data, and (2) unable to support decision trees. Existing FL solutions for vertically partitioned data or decision trees require heavy cryptographic operations.In this paper, we propose a framework named FederBoost for private federated learning of gradient boosting decision trees (GBDT). It supports running GBDT over both horizontally and vertically partitioned data. The key observation for designing FederBoost is that the whole training process of GBDT relies on the order of the data instead of the values. Consequently, vertical FederBoost does not require any cryptographic operation and horizontal FederBoost only requires lightweight secure aggregation.We fully implement FederBoost and evaluate its utility and efficiency through extensive experiments performed on three public datasets. Our experimental results show that both vertical and horizontal FederBoost achieve the same level of AUC with centralized training where all data are collected in a central server; and both of them can finish training within half an hour even in WAN.

A Fair and Efficient Hybrid Federated Learning Framework based on XGBoost for Distributed Power Prediction

“…To this end, we propose a hybrid XGBoost-based federated learning framework. In Framework 2, we seamlessly integrate FederBoost [27] with SecureBoost [28], which are respectively a horizontal and vertical federated learning framework, to address both types of data separation. In this framework, we denote C m as the list of label-holding parties (i.e.…”

“…For horizontally-separated data across districts, the edges of bins in XGBoost are dependent on the statistic information of all homogeneous parties. Therefore, we resort to FederBoost's secure bin construction algorithm [27] to find proper bin edges. For the label-holding parties, the active party C m A leads the bin construction process on features of all C m s; the same goes for secondary parties.…”

“…Li et al proposed a serverless framework [26] where each party boosted a number of trees in turn, by exploiting similarity information based on locality-sensitive hashing. Tian et al proposed a horizontal and serverless FederBoost framework for private federated learning of gradient boosting decision trees [27], in which all the uploaded gradients were transferred to the active party for node splitting, and only a lightweight secure aggregation scheme was required. The essence of gradient aggregation in these frameworks, if properly combined with an XGBoostbased vertical framework such as SecureBoost [28], can result in a hybrid federated learning framework entirely based on XGBoost, to address the case where data are both horizontally and vertically separated.…”

Communication Efficiency in Federated Learning: Achievements and Challenges

“…This is the original sense of FL learning where data from each domain is homogeneous, see in figure 3, and contributing together to train the global ML model together [25]. This can be explained using the original example that is presented by Google [6] wherein the global model is an aggregate of locally trained multiple participating devices [27].…”

Fed-GBM

“…They introduce extra computation cost in model training and degrade the model performance at runtime. In addition, communication efficiency is usually neglected in recent solutions [30,46,49], which, however, is a noted and important factor in designing a collaborative learning system. Large communication footprints could hamper the system scalability and cause network congestion, particularly when bandwidth is limited at the user end.…”