CodedPrivateML: A Fast and Privacy-Preserving Framework for Distributed Machine Learning

So, Jinhyun; Güler, Başak; Avestimehr, A. Salman; Mohassel, Payman

doi:10.48550/arxiv.1902.00641

Cited by 25 publications

(19 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Canoe contributes to such approaches by proposing a generic system support that makes them applicable to scenarios based on neural network models. Other types of distributed machine learning system introduce complementary techniques aimed at model specializations [69,85,88], including when targeting edge nodes and IoT, as well as for addressing privacy concerns about data and model sharing [42,59,77]. Knowledge Transfer Technique.…”

Section: Related Workmentioning

confidence: 99%

Canoe : A System for Collaborative Learning for Neural Nets

Daga¹,

Chen²,

Agrawal³

et al. 2021

Preprint

View full text Add to dashboard Cite

For highly distributed environments such as edge computing, collaborative learning approaches eschew the dependence on a global, shared model, in favor of models tailored for each location. Creating tailored models for individual learning contexts reduces the amount of data transfer, while collaboration among peers provides acceptable model performance. Collaboration assumes, however, the availability of knowledge transfer mechanisms, which are not trivial for deep learning models where knowledge isn't easily attributed to precise model slices. We present Canoe -a framework that facilitates knowledge transfer for neural networks. Canoe provides new system support for dynamically extracting significant parameters from a helper node's neural network, and uses this with a multi-model boosting-based approach to improve the predictive performance of the target node. The evaluation of Canoe with different PyTorch and TensorFlow neural network models demonstrates that the knowledge transfer mechanism improves the model's adaptiveness to changes up to 3.5× compared to learning in isolation, while affording several magnitudes reduction in data movement costs compared to federated learning.

show abstract

Section: Related Workmentioning

confidence: 99%

Canoe : A System for Collaborative Learning for Neural Nets

Daga¹,

Chen²,

Agrawal³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In Step 1, given any upper bound construction of R(p, m, n) (e.g., Strassen's construction) with rank R and tensor tuples a ∈ F R×p×m , b ∈ F R×p×n , and c ∈ F R×m×n , we pre-encode the inputs each into a list of R coded submatrices. 7 Ãi,vec…”

Section: Achievability Schemes For Secure Distributed Matrix Multipli...mentioning

confidence: 99%

“…Large scale distributed computing faces several modern challenges, in particular, to provide resiliency against stragglers, robustness against computing errors, security against Byzantine and eavesdropping adversaries, privacy of sensitive information, and to efficiently handle repetitive computation [1]- [7]. Coded computing is an emerging field that resolves these issues by introducing and developing new coding theoretic concepts, started focusing on straggler mitigation [8]-[10], then later extended to secure and private computation [6], [7], [11]- [14].…”

Section: Introductionmentioning

confidence: 99%

Entangled Polynomial Codes for Secure, Private, and Batch Distributed Matrix Multiplication: Breaking the "Cubic" Barrier

Avestimehr

2020

Preprint

Self Cite

View full text Add to dashboard Cite

In distributed matrix multiplication, a common scenario is to assign each worker a fraction of the multiplication task, by partition the input matrices into smaller submatrices. In particular, by dividing two input matrices into m-by-p and p-by-n subblocks, a single multiplication task can be viewed as computing linear combinations of pmn submatrix products, which can be assigned to pmn workers. Such block-partitioning based designs have been widely studied under the topics of secure, private, and batch computation, where the state of the arts all require computing at least "cubic" (pmn) number of submatrix multiplications. Entangled polynomial codes, first presented for straggler mitigation, provides a powerful method for breaking the cubic barrier. It achieves a subcubic recovery threshold, meaning that the final product can be recovered from any subset of multiplication results with a size order-wise smaller than pmn. In this work, we show that entangled polynomial codes can be further extended to also include these three important settings, and provide a unified framework that order-wise reduces the total computational costs upon the state of the arts by achieving subcubic recovery thresholds. I. INTRODUCTIONLarge scale distributed computing faces several modern challenges, in particular, to provide resiliency against stragglers, robustness against computing errors, security against Byzantine and eavesdropping adversaries, privacy of sensitive information, and to efficiently handle repetitive computation [1]- [7]. Coded computing is an emerging field that resolves these issues by introducing and developing new coding theoretic concepts, started focusing on straggler mitigation [8]-[10], then later extended to secure and private computation [6], [7], [11]- [14].Coding for straggler mitigation is first studied in [8] for linear computation, where classical linear codes can be directly applied to achieve same performances. For computation beyond linear functions, new classes of coding designs are needed to achieve optimality. In [10], we studied matrix-by-matrix multiplication and introduced the polynomial coded computing (PCC) framework. The main coding idea is to jointly encode the input variables into single variate polynomials where the coefficients are functions of the inputs. By assigning each worker evaluations of these polynomials as coded variables, they essentially evaluate a new polynomial composed by each worker's computation and the encoding functions at the same point. As long as the needed final results can be recovered from the coefficients of the composed polynomial, the master can decode the final output when sufficiently many workers complete their computation. PCC significantly reduces the design problem of coded computation to finding polynomials satisfying the above decodability constraint while minimizing the degree of the decomposed polynomial. It has been shown that PCC achieves a great success in providing exact optimal coding constructions for large classes of computation tasks includi...

show abstract

“…Another closely related line of work is concerned with individual samples' privacy; this is relevant in cases where the samples contain highly sensitive information and they need to be kept secret even from the ML model. Examples of privacy-preserving ML include works based on differential privacy such as [22][23][24] and privacy-preserving learning such as [25][26][27][28][29][30]. A major distinction between this line of work and machine unlearning is that samples do not need to be kept private in machine unlearning, but the requests of unlearning need to be honored.…”

Section: Related Workmentioning

confidence: 99%

Coded Machine Unlearning

Aldaghri,

Mahdavifar,

Beirami

2020

Preprint

View full text Add to dashboard Cite

Models trained in machine learning processes may store information about individual samples used in the training process. There are many cases where the impact of an individual sample may need to be deleted and unlearned (i.e., removed) from the model. Retraining the model from scratch after removing a sample from its training set guarantees perfect unlearning, however, it becomes increasingly expensive as the size of training dataset increases. One solution to this issue is utilizing an ensemble learning method that splits the dataset into disjoint shards and assigns them to non-communicating weak learners and then aggregates their models using a pre-defined rule. This framework introduces a trade-off between performance and unlearning cost which may result in an unreasonable performance degradation, especially as the number of shards increases. In this paper, we present a coded learning protocol where the dataset is linearly coded before the learning phase. We also present the corresponding unlearning protocol for the aforementioned coded learning model along with a discussion on the proposed protocol's success in ensuring perfect unlearning. Finally, experimental results show the effectiveness of the coded machine unlearning protocol in terms of performance versus unlearning cost trade-off.

show abstract

CodedPrivateML: A Fast and Privacy-Preserving Framework for Distributed Machine Learning

Cited by 25 publications

References 0 publications

Canoe : A System for Collaborative Learning for Neural Nets

Canoe : A System for Collaborative Learning for Neural Nets

Entangled Polynomial Codes for Secure, Private, and Batch Distributed Matrix Multiplication: Breaking the "Cubic" Barrier

Coded Machine Unlearning

Contact Info

Product

Resources

About