A Principled Approach to Data Valuation for Federated Learning

Abstract

Federated Learning (FL), a distributed learning paradigm that scales on-device learning collaboratively, has emerged as a promising approach for decentralized AI applications. Local optimization methods such as Federated Averaging (FedAvg) are the most prominent methods for FL applications. Despite their simplicity and popularity, the theoretical understanding of local optimization methods is far from clear. This dissertation aims to advance the theoretical foundation of local methods in the following three directions.First, we establish sharp bounds for FedAvg, the most popular algorithm in Federated Learning. We demonstrate how FedAvg may suffer from a notion we call iterate bias, and how an additional third-order smoothness assumption may mitigate this effect and lead to better convergence rates. We explain this phenomenon from a Stochastic Differential Equation (SDE) perspective.Second, we propose Federated Accelerated Stochastic Gradient Descent (FedAc), the first principled acceleration of FedAvg, which provably improves the convergence rate and communication efficiency.Our technique uses on a potential-based perturbed iterate analysis, a novel stability analysis of generalized accelerated SGD, and a strategic tradeoff between acceleration and stability.Third, we study the Federated Composite Optimization problem, which extends the classic smooth setting by incorporating a shared non-smooth regularizer. We show that direct extensions of FedAvg may suffer from the "curse of primal averaging," resulting in slow convergence. As a solution, we propose a new primal-dual algorithm, Federated Dual Averaging, which overcomes the curse of primal averaging by employing a novel inter-client dual averaging procedure.1. Establishing sharp understanding of the existing FL algorithms.

GTG-Shapley: Efficient and Accurate Participant Contribution Evaluation in Federated Learning

“…To leverage SV for measuring FL participant contributions, current research focuses is on improving the eiciency of SV approximation while maintaining accuracy. Existing works in this domain can be divided into two main categories: 1) reducing the number of sub-model evaluations required [7,9,25], and 2) accelerating single round of evaluation [20,25]. To accelerate a single round of evaluation, FL can utilize participants' gradient updates to reconstruct various FL sub-models instead of re-training each variant of the model from scratch.…”

“…However, they still require evaluating every possible FL sub-model as in the original SV setting. In [25], the authors proposed the Federated SV approach with two acceleration techniques for within-round SV estimation: 1) random permutation sampling and 2) group testing. However, the number of random sampling is ixed and not tailored to each permutation.…”

“…Evaluating participants' contributions under the canonical SV settings requires retraining the FL model for each combination of participants [2,7,19], which is computationally prohibitive. Replacing FL sub-model evaluation to gradient-based FL model reconstruction [20,25] is necessary.…”

“…The inal SV for a participant is the sum of his SVs in all rounds. (5) Fed-SV: This method [25] approximates the 'federated Shapley value' by group testing-based estimation as GroupTesting. The diferences are 1) the subsets performances used for estimating Shapley diferences are evaluated on sub-model reconstructed by participants model parameters, and 2) SVs are estimated independently each round and aggregated later.…”

In the Pursuit of Privacy: The Promises and Predicaments of Federated Learning in Healthcare

“…Within a FL framework, the unique nature of having the data separated entails concern about an equitable distribution of profit. Shapley values are a wellestablished method for determination of equitable payoff, yet in FL they pose a computationally prohibitive and horizontally limited applicability due in part to its lack of ability to be extrapolated (Huang et al, 2020;Wang et al, 2020). Yet, the nature of FL does warrant an evaluation of the utility it poses as well as any derivative models that can be created such as FedCoin (Liu et al, 2020).…”