Joint Parameter-and-Bandwidth Allocation for Improving the Efficiency of Partitioned Edge Learning

Abstract

To leverage data and computation capabilities of mobile devices, machine learning algorithms are deployed at the network edge for training artificial intelligence (AI) models, resulting in the new paradigm of edge learning. In this paper, we consider the framework of partitioned edge learning for iteratively training a large-scale model using many resource-constrained devices (called workers). To this end, in each iteration, the model is dynamically partitioned into parametric blocks, which are downloaded to worker groups for updating using data subsets. Then, the local updates are uploaded to and cascaded by the server for updating a global model. To reduce resource usage by minimizing the total learning-and-communication latency, this work focuses on the novel joint design of parameter (computation load) allocation and bandwidth allocation (for downloading and uploading). Two design approaches are adopted. First, a practical sequential approach, called partially integrated parameter-andbandwidth allocation (PABA), yields two schemes, namely bandwidth aware parameter allocation and parameter aware bandwidth allocation. The former minimizes the load for the slowest (in computing) of worker groups, each training a same parametric block. The latter allocates the largest bandwidth to the worker being the latency bottleneck. Second, PABA are jointly optimized. Despite it being a nonconvex problem, an efficient and optimal solution algorithm is derived by intelligently nesting a bisection search and solving a convex problem. Experimental results using real data demonstrate that integrating PABA can substantially improve the performance of partitioned edge learning in terms of latency (by e.g., 46%) and accuracy (by e.g., 4% given the latency of 100 seconds).

Accelerating Partitioned Edge Learning via Joint Parameter-and-Bandwidth Allocation

“…One is the number of communication rounds required for model convergence and the other is the per-round latency. According to [13], the overall learning latency minimization is equivalent to separately minimizing the per-round latency, as the number of rounds till model convergence is irrelevant to the parameter and bandwidth allocation. In the sequel, the per-round latency minimization problem is formulated.…”

“…where the notation follows that in Problem (P3). Two useful Lemmas for relating Problems (P3) and (P4) are given as follows, whose proofs can be referred to Appendices F and G in [13].…”

“…One is the baseline, where the number of parameters assigned to one worker is proportional to its computation capability and bandwidths are allocated equally. Other two algorithms are parameter aware bandwidth allocation in Section IV and bandwidth aware parameter allocation in [13].…”

“…where Tph andTk,n =tk,n(b * k ) are constants. In (13), tk,n (ρk,n) is a convex function of ρk,n. Then, the objective function of (P2)is also convex, as max operation preserves convexity.…”

Adaptive Subcarrier, Parameter, and Power Allocation for Partitioned Edge Learning Over Broadband Channels

“…Moreover, researchers have designed energy efficient RRM techniques to tackle the challenge of executing a complex learning task at energy constrained devices in a FEEL system [18]- [20]. Recently, researchers have also explored the efficient implementation of parameter-server training over wireless channels, resulting in a framework called partitioned edge learning (PARTEL) [21]. Let parameter allocation refers to the system operation that to balance computation loads, the server divides the model into parametric blocks of variable lengths and allocate them to devices for separate training.…”