Evaluation and Optimization of Distributed Machine Learning Techniques for Internet of Things

Gao, Yansong; Kim, Minki; Thapa, Chandra; Abuadbba, Alsharif; Zhang, Zhi; Camtepe, Seyit; Kim, Hyoungshick; Nepal, Surya

doi:10.1109/tc.2021.3135752

Cited by 43 publications

(29 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These operations are not affected by the client order as the local client-side models are aggregated by the weighted averaging method, i.e., FedAvg. Some other SFL versions are available in the literature, but they are developed after and influenced by our approach (Han, amd Jungmoon Lee, and Moon 2021; Gao et al 2021).…”

Section: Variants Of Splitfedmentioning

confidence: 99%

SplitFed: When Federated Learning Meets Split Learning

Thapa

Chamikara

Camtepe

et al. 2022

AAAI

Self Cite

281

View full text Add to dashboard Cite

Federated learning (FL) and split learning (SL) are two popular distributed machine learning approaches. Both follow a model-to-data scenario; clients train and test machine learning models without sharing raw data. SL provides better model privacy than FL due to the machine learning model architecture split between clients and the server. Moreover, the split model makes SL a better option for resource-constrained environments. However, SL performs slower than FL due to the relay-based training across multiple clients. In this regard, this paper presents a novel approach, named splitfed learning (SFL), that amalgamates the two approaches eliminating their inherent drawbacks, along with a refined architectural configuration incorporating differential privacy and PixelDP to enhance data privacy and model robustness. Our analysis and empirical results demonstrate that (pure) SFL provides similar test accuracy and communication efficiency as SL while significantly decreasing its computation time per global epoch than in SL for multiple clients. Furthermore, as in SL, its communication efficiency over FL improves with the number of clients. Besides, the performance of SFL with privacy and robustness measures is further evaluated under extended experimental settings.

show abstract

Section: Variants Of Splitfedmentioning

confidence: 99%

SplitFed: When Federated Learning Meets Split Learning

Thapa

Chamikara

Camtepe

et al. 2022

AAAI

Self Cite

281

View full text Add to dashboard Cite

show abstract

“…The Conv1-D architecture [ 28 ] was used for the primary experimentation of the ECG time-series dataset. Regarding architecture selection for ECG, the Conv1-D architecture was selected because of its efficiency in dealing with sequential data [ 29 , 30 ]. Another determining factor for selecting the Conv1-D architecture instead of sequential models (such as LSTM, GRU, and RNN) is that there is no effective approach for splitting the sequence model between the client and server in the SFL setting.…”

Section: Datasets and Modelsmentioning

confidence: 99%

Performance and Information Leakage in Splitfed Learning and Multi-Head Split Learning in Healthcare Data and Beyond

Joshi

Thapa

Camtepe

et al. 2022

MPs

Self Cite

View full text Add to dashboard Cite

Machine learning (ML) in healthcare data analytics is attracting much attention because of the unprecedented power of ML to extract knowledge that improves the decision-making process. At the same time, laws and ethics codes drafted by countries to govern healthcare data are becoming stringent. Although healthcare practitioners are struggling with an enforced governance framework, we see the emergence of distributed learning-based frameworks disrupting traditional-ML-model development. Splitfed learning (SFL) is one of the recent developments in distributed machine learning that empowers healthcare practitioners to preserve the privacy of input data and enables them to train ML models. However, SFL has some extra communication and computation overheads at the client side due to the requirement of client-side model synchronization. For a resource-constrained client side (hospitals with limited computational powers), removing such conditions is required to gain efficiency in the learning. In this regard, this paper studies SFL without client-side model synchronization. The resulting architecture is known as multi-head split learning (MHSL). At the same time, it is important to investigate information leakage, which indicates how much information is gained by the server related to the raw data directly out of the smashed data—the output of the client-side model portion—passed to it by the client. Our empirical studies examine the Resnet-18 and Conv1-D architecture model on the ECG and HAM-10000 datasets under IID data distribution. The results find that SFL provides 1.81% and 2.36% better accuracy than MHSL on the ECG and HAM-10000 datasets, respectively (for cut-layer value set to 1). Analysis of experimentation with various client-side model portions demonstrates that it has an impact on the overall performance. With an increase in layers in the client-side model portion, SFL performance improves while MHSL performance degrades. Experiment results also demonstrate that information leakage provided by mutual information score values in SFL is more than MHSL for ECG and HAM-10000 datasets by 2×10−5 and 4×10−3, respectively.

show abstract

“…Therefore, [10] decided to deploy edge servers as coadjutants to alleviate the communication and computation load of the SL server; then each edge server can interact with one or several clients to exchange gradients; while the SL server can further calculate averaged gradients and update the subnetworks at edge servers. [11] put forward similar ideas by deploying multiple FL servers to handle groups of clients. Additionally, they have made comprehensive experiments on Raspberry Pi devices.…”

Section: Related Workmentioning

confidence: 99%

“…For cellular clients, we borrow ideas from [8]- [11], and integrate them into the proposed HFSL. As shown by the right side of Fig.…”

Section: B Architecture Of Hfslmentioning

confidence: 99%

“…In the next training round, the training starts with different clients under a new model splitting setting. Apparently, the training process is sequential within each D2D client cluster, and parallel over different clusters.For cellular clients, we borrow ideas from[8]-[11], and integrate them into the proposed HFSL. As shown by the right side of Fig.2(b), the ML model is split into two subnetworks C and H, where each client trains the same subnetwork C in parallel while the subnetwork H is deployed at edge server B.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Hybrid Architectures for Distributed Machine Learning in Heterogeneous Wireless Networks

Cheng¹,

Fan²,

Liwang³

et al. 2022

Preprint

View full text Add to dashboard Cite

The ever-growing data privacy concerns have transformed machine learning (ML) architectures from centralized to distributed, leading to federated learning (FL) and split learning (SL) as the two most popular privacy-preserving ML paradigms. However, implementing either conventional FL or SL alone with diverse network conditions (e.g., device-to-device (D2D) and cellular communications) and heterogeneous clients (e.g., heterogeneous computation/communication/energy capabilities) may face significant challenges, particularly poor architecture scalability and long training time. To this end, this article proposes two novel hybrid distributed ML architectures, namely, hybrid split FL (HSFL) and hybrid federated SL (HFSL), by combining the advantages of both FL and SL in D2D-enabled heterogeneous wireless networks. Specifically, the performance comparison and advantages of HSFL and HFSL are analyzed generally. Promising open research directions are presented to offer commendable reference for future research. Finally, primary simulations are conducted upon considering three datasets under non-independent and identically distributed settings, to verify the feasibility of our proposed architectures, which can significantly reduce communication/computation cost and training time, as compared with conventional FL and SL.

show abstract

Evaluation and Optimization of Distributed Machine Learning Techniques for Internet of Things

Cited by 43 publications

References 25 publications

SplitFed: When Federated Learning Meets Split Learning

SplitFed: When Federated Learning Meets Split Learning

Performance and Information Leakage in Splitfed Learning and Multi-Head Split Learning in Healthcare Data and Beyond

Hybrid Architectures for Distributed Machine Learning in Heterogeneous Wireless Networks

Contact Info

Product

Resources

About