Communication-Adaptive Stochastic Gradient Methods for Distributed Learning

Chen, Tianyi; Sun, Yuejiao; Yin, Wotao

doi:10.1109/tsp.2021.3099977

Cited by 15 publications

(16 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As shown in Corollary 1, the overall sample complexity (i.e., the total number of data samples required to achieve an -accurate stationary point) of our FBO-AggITD is O(κ 9 −2 ), which matches the sample complexities of stocBiO (Ji et al, 2021), BSA (Ghadimi & Wang, 2018) and ALSET (Chen et al, 2021a) in the non-federated bilevel optimization and FedNest (Tarzanagh et al, 2022) in the federated setting despite the data heterogeneity. Compared to FedNest, our method is much simpler with lower communication costs.…”

Section: Convergence and Complexity Analysissupporting

confidence: 66%

“…), as shown in the final convergence analysis. It is worth mentioning that these two terms match the error bound of the stochastic AID-based hypergradient estimator in non-federated setting (Ji et al, 2021;Ghadimi & Wang, 2018;Chen et al, 2021a), and hence our analysis can be of independent interest to non-federated bilevel optimization. Also note that the last two non-vanishing error terms O(λ 2 β 2 ) and O(λβ 2 ) are induced by the client drift in the y updates, which exists specially in the FL, can be addressed by choosing a sufficiently small stepsize β. Technically, we first shows via a recursive analysis that the key approximation error between the expected indirect part of the AggITD estimator…”

Section: Estimation Properties For Aggitdmentioning

confidence: 64%

“…end for 7: end for 8: Hong et al, 2020;Khanduri et al, 2021;Chen et al, 2021a) as well as in federated bilevel optimization (Tarzanagh et al, 2022).…”

Section: Definitions and Assumptionsmentioning

confidence: 99%

“…( 10), it is noted that the hypergradient ∇f (x) = ∇f (x, y * (x) ). By the analysis in Ghadimi &Chen et al 2021a, the following lemma characterizes the continuity and smoothness properties of the inner and outer functions (f i , g i ) for all i ∈ S.…”

Section: B Notationsmentioning

confidence: 99%

See 3 more Smart Citations

Communication-Efficient Federated Hypergradient Computation via Aggregated Iterative Differentiation

Xiao¹,

Ji²

2023

Preprint

View full text Add to dashboard Cite

Federated bilevel optimization has attracted increasing attention due to emerging machine learning and communication applications. The biggest challenge lies in computing the gradient of the upper-level objective function (i.e., hypergradient) in the federated setting due to the nonlinear and distributed construction of a series of global Hessian matrices. In this paper, we propose a novel communication-efficient federated hypergradient estimator via aggregated iterative differentiation (AggITD). AggITD is simple to implement and significantly reduces the communication cost by conducting the federated hypergradient estimation and the lower-level optimization simultaneously. We show that the proposed AggITD-based algorithm achieves the same sample complexity as existing approximate implicit differentiation (AID)based approaches with much fewer communication rounds in the presence of data heterogeneity. Our results also shed light on the great advantage of ITD over AID in the federated/distributed hypergradient estimation. This differs from the comparison in the non-distributed bilevel optimization, where ITD is less efficient than AID. Our extensive experiments demonstrate the great effectiveness and communication efficiency of the proposed method.

show abstract

Section: Convergence and Complexity Analysissupporting

confidence: 66%

Section: Estimation Properties For Aggitdmentioning

confidence: 64%

“…end for 7: end for 8: Hong et al, 2020;Khanduri et al, 2021;Chen et al, 2021a) as well as in federated bilevel optimization (Tarzanagh et al, 2022).…”

Section: Definitions and Assumptionsmentioning

confidence: 99%

Section: B Notationsmentioning

confidence: 99%

See 2 more Smart Citations

Communication-Efficient Federated Hypergradient Computation via Aggregated Iterative Differentiation

Xiao¹,

Ji²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, several prevailing machine learning applications can be naturally formulated as a bilevel programming problem (Maclaurin et al, 2015;Pedregosa, 2016;Finn et al, 2017;Franceschi et al, 2017Franceschi et al, , 2018Ji et al, 2020), which brings a lot of attention to the bilevel programming in the machine learning community. On the theoretical side, there are many existing works deriving both asymptotic (Franceschi et al, 2018;Shaban et al, 2019;Liu et al, 2021) and non-asymptotic (Ghadimi & Wang, 2018;Hong et al, 2020;Chen et al, 2021a;Guo & Yang, 2021; convergence analysis for the determinstic or stochastic bilevel optimization. For example, Ghadimi & Wang 2018;Hong et al 2020;Arbel & Mairal 2022 proved the convergence for SGD type of bilevel methods via the approximate implicit differentiation (AID) approach.…”

Section: Related Workmentioning

confidence: 99%

LoAdaBoost: Loss-based AdaBoost federated machine learning with reduced computational complexity on IID and non-IID intensive care data

et al. 2020

View full text Add to dashboard Cite

Intensive care data are valuable for improvement of health care, policy making and many other purposes. Vast amount of such data are stored in different locations, on many different devices and in different data silos. Sharing data among different sources is a big challenge due to regulatory, operational and security reasons. One potential solution is federated machine learning, which is a method that sends machine learning algorithms simultaneously to all data sources, trains models in each source and aggregates the learned models. This strategy allows utilization of valuable data without moving them. One challenge in applying federated machine learning is the possibly different distributions of data from diverse sources. To tackle this problem, we proposed an adaptive boosting method named LoAda-Boost that increases the efficiency of federated machine learning. Using intensive care unit data from hospitals, we investigated the performance of learning in IID and non-IID data distribution scenarios, and showed that the proposed LoAdaBoost method achieved higher predictive accuracy with lower computational complexity than the baseline method.

show abstract