Can we Generalize and Distribute Private Representation Learning?

Azam, Sheikh Shams; Kim, Tae-Jin; Hosseinalipour, Seyyedali; Joe‐Wong, Carlee; Bagchi, Saurabh; Brinton, Christopher G.

doi:10.48550/arxiv.2010.01792

Cited by 2 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…n /D (k) . Then, the cumulative average of the global loss function gradient for PSL satisfies (22). Proof.…”

Section: T)(26)mentioning

confidence: 99%

“…Also, economic incentives (e.g., rewards, gas credit, or cash back) can also facilitate data sharing in many applications such as autonomous driving. Further, there exists emerging research on privacy preserving representation learning, which aims to obfuscate the sensitive attributes of data, making the data shareable without privacy concerns [21], [22]. These will make the data sharing a viable method in the near future.…”

Section: B Fedl Shortcomings and Solution Overviewmentioning

confidence: 99%

“…Bound (22) no longer depends on consecutive loss gains ((a) in ( 21)), rather it depends on the initial error F (−1) (w (0) ) − F (K) . We next obtain the conditions under which the cumulative average of the global gradient converges to zero: Corollary 2 (Convergence under Unified Upperbounds on the Sampling Noise).…”

Section: T)(26)mentioning

confidence: 99%

See 2 more Smart Citations

From Federated to Fog Learning: Distributed Machine Learning over Heterogeneous Wireless Networks

et al. 2020

Self Cite

View full text Add to dashboard Cite

Federated learning (FedL) has emerged as a popular technique for distributing model training over a set of wireless devices, via iterative local updates (at devices) and global aggregations (at the server). In this paper, we develop parallel successive learning (PSL), which expands the FedL architecture along three dimensions: (i) Network, allowing decentralized cooperation among the devices via device-to-device (D2D) communications. (ii) Heterogeneity, interpreted at three levels: (ii-a) Learning: PSL considers heterogeneous number of stochastic gradient descent iterations with different mini-batch sizes at the devices; (ii-b) Data: PSL presumes a dynamic environment with data arrival and departure, where the distributions of local datasets evolve over time, captured via a new metric for model/concept drift. (ii-c) Device: PSL considers devices with different computation and communication capabilities. (iii) Proximity, where devices have different distances to each other and the access point. PSL considers the realistic scenario where global aggregations are conducted with idle times in-between them for resource efficiency improvements, and incorporates data dispersion and model dispersion with local model condensation into FedL. Our analysis sheds light on the notion of cold vs. warmed up models, and model inertia in distributed machine learning. We then propose networkaware dynamic model tracking to optimize the model learning vs. resource efficiency tradeoff, which we show is an NP-hard signomial programming problem. We finally solve this problem through proposing a general optimization solver. Our numerical results reveal new findings on the interdependencies between the idle times in-between the global aggregations, model/concept drift, and D2D cooperation configuration.

show abstract

“…n /D (k) . Then, the cumulative average of the global loss function gradient for PSL satisfies (22). Proof.…”

Section: T)(26)mentioning

confidence: 99%

Section: B Fedl Shortcomings and Solution Overviewmentioning

confidence: 99%

See 1 more Smart Citation

From Federated to Fog Learning: Distributed Machine Learning over Heterogeneous Wireless Networks

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…Another closely related line of work is concerned with individual samples' privacy; this is relevant in cases where the samples contain highly sensitive information and they need to be kept secret even from the ML model. Examples of privacy-preserving ML include works based on differential privacy such as [22][23][24] and privacy-preserving learning such as [25][26][27][28][29][30]. A major distinction between this line of work and machine unlearning is that samples do not need to be kept private in machine unlearning, but the requests of unlearning need to be honored.…”

Section: Related Workmentioning

confidence: 99%

Coded Machine Unlearning

Aldaghri,

Mahdavifar,

Beirami

2020

Preprint

View full text Add to dashboard Cite

Models trained in machine learning processes may store information about individual samples used in the training process. There are many cases where the impact of an individual sample may need to be deleted and unlearned (i.e., removed) from the model. Retraining the model from scratch after removing a sample from its training set guarantees perfect unlearning, however, it becomes increasingly expensive as the size of training dataset increases. One solution to this issue is utilizing an ensemble learning method that splits the dataset into disjoint shards and assigns them to non-communicating weak learners and then aggregates their models using a pre-defined rule. This framework introduces a trade-off between performance and unlearning cost which may result in an unreasonable performance degradation, especially as the number of shards increases. In this paper, we present a coded learning protocol where the dataset is linearly coded before the learning phase. We also present the corresponding unlearning protocol for the aforementioned coded learning model along with a discussion on the proposed protocol's success in ensuring perfect unlearning. Finally, experimental results show the effectiveness of the coded machine unlearning protocol in terms of performance versus unlearning cost trade-off.

show abstract

Can we Generalize and Distribute Private Representation Learning?

Cited by 2 publications

References 0 publications

From Federated to Fog Learning: Distributed Machine Learning over Heterogeneous Wireless Networks

From Federated to Fog Learning: Distributed Machine Learning over Heterogeneous Wireless Networks

Coded Machine Unlearning

Contact Info

Product

Resources

About