Training Time Minimization for Federated Edge Learning with Optimized Gradient Quantization and Bandwidth Allocation

Abstract

Training a machine learning model with federated edge learning (FEEL) is typically time-consuming due to the constrained computation power of edge devices and limited wireless resources in edge networks. In this paper, the training time minimization problem is investigated in a quantized FEEL system, where the heterogeneous edge devices send quantized gradients to the edge server via orthogonal channels. In particular, a stochastic quantization scheme is adopted for compression of uploaded gradients, which can reduce the burden of per-round communication but may come at the cost of increasing number of communication rounds. The training time is modeled by taking into account the communication time, computation time and the number of communication rounds. Based on the proposed training time model, the intrinsic trade-off between the number of communication rounds and per-round latency is characterized. Specifically, we analyze the convergence behavior of the quantized FEEL in terms of the optimality gap. Further, a joint dataand-model-driven fitting method is proposed to obtain the exact optimality gap, based on which the closed-form expressions for the number of communication rounds and the total training time are obtained. Constrained by total bandwidth, the training time minimization problem is formulated as a joint quantization level and bandwidth allocation optimization problem. To this end, an algorithm based on alternating optimization is proposed, which alternatively solves the subproblem of quantization optimization via successive convex approximation and the subproblem of bandwidth allocation via bisection search. With different learning tasks and models, the validation of our analysis and the nearoptimal performance of the proposed optimization algorithm are demonstrated by the experimental results.

A Survey of State-of-the-art on Edge Computing: Theoretical Models, Technologies, Directions, and Development Paths

“…The results of simulations on Jetson TX2 show that this scheme improves DL inference throughput by a factor of 3.3 to 3.8. Liu et al [104] proposed a training time model and an alternating optimization-based algorithm to solve the training time minimization problem in the quantized FEEL system. Experiments show that the optimization algorithm proposed by the authors can approach the optimal performance under different learning tasks and models.…”

Adaptive Quantization based on Ensemble Distillation to Support FL enabled Edge Intelligence

“…In general, the original local model consists of the training weights that are float 32bit, which may be quantized to integer c-bit (1 ≤ c < 32) quantized model [2]. Let b(Q c t (w i,r u ) = (1 + log 2 (c + 1))|w i,r u | represent the volume of the transmitted quantized model of n i u , which is a function of the size of the quantized weights (i.e., |w i,r u |) as well as the bit width c [9]. Therefore, based on Shannon's theorem, the wireless bandwidth used to transmit the local quantized model of UE n i u during the r−th communication round can be given by…”

“…Note that problem (8) could be equivalent to problem (9), as we can always find the available multipliers λ 1 and λ 2 for problem (9) to approximate the optimal solution of problem (8) [12]. In other words, we can find the optimal solution of problem (8) via problem (9).…”

“…Note that problem (8) could be equivalent to problem (9), as we can always find the available multipliers λ 1 and λ 2 for problem (9) to approximate the optimal solution of problem (8) [12]. In other words, we can find the optimal solution of problem (8) via problem (9). In this paper, with the aim to obtain an adaptive FL quantized model based on the available bandwidth resources and channel quality while guaranteeing learning performance among heterogeneous models, we propose AQeD scheme based on FL to solve problem (9) by introducing a number of local and global iterations.…”