DISTREAL: Distributed Resource-Aware Learning in Heterogeneous Systems

Abstract

We study the problem of distributed training of neural networks (NNs) on devices with heterogeneous, limited, and time-varying availability of computational resources. We present an adaptive, resource-aware, on-device learning mechanism, DISTREAL, which is able to fully and efficiently utilize the available resources on devices in a distributed manner, increasing the convergence speed. This is achieved with a dropout mechanism that dynamically adjusts the computational complexity of training an NN by randomly dropping filters of convolutional layers of the model. Our main contribution is the introduction of a design space exploration (DSE) technique, which finds Pareto-optimal per-layer dropout vectors with respect to resource requirements and convergence speed of the training. Applying this technique, each device is able to dynamically select the dropout vector that fits its available resource without requiring any assistance from the server. We implement our solution in a federated learning (FL) system, where the availability of computational resources varies both between devices and over time, and show through extensive evaluation that we are able to significantly increase the convergence speed over the state of the art without compromising on the final accuracy.

Federated Learning for Computationally Constrained Heterogeneous Devices: A Survey

“…Masks are updated every round. In DISTREAL [80], the authors explore how subsets can be trained in environments with time-varying computational resources that change faster than FL rounds and are not known in advance. A mini-batch level granularity for training subsets by randomly switching filters of the CNN during training is achieved.…”

“…The attributes scale and granularity are often neglected, are hidden behind the technique, and lack discussion in the papers. The reported scale in the resources supported by the techniques ranges from 4× − 25× [12,41,52,61,71,77,79,80,85,101] up to 100× − 250× [25,87], yet it remains unclear whether training at such high scales is still effective. Hence, while all approaches show the effectiveness of their solution in certain scenarios, it often remains unclear whether devices with low resources or stale devices can make a meaningful contribution that advances the global model.…”

“…Especially in iid settings, current state-of-the-art works do not compare themselves against trivial baselines such as dropping of devices (accepting a smaller total share of data), which is the solution that current real-world FL applications like Google GBoard employ. A second trivial baseline is deploying a low-complexity model for all devices [80], which can already outperform some state-of-the-art techniques. A potential solution to maintaining effectiveness for large-scale heterogeneity with fine-grained granularity could be the interplay of system-level and NN-level approaches, as their favorable properties could complement each other.…”

“…the resource model. This is especially the case in techniques using subsets, where some model resources in terms of power usage [104], while others count the number of parameters [2,25,41,79], the number of multiplyaccumulate-operations [41,80], or the required training time [77]. Therefore, the supported scale of heterogeneity by the techniques is not comparable.…”

Towards AI-Native Vehicular Communications

“…Other solutions entail work from the clients' side. Under a known time limitations, workers either speed up local training by allocating more computing resources or training models partially [27] with gradient sparsification [28], or reduce the communication time by compressing the model [29]. Such methods do not apply in those conditions in which vehicles completely lose the connection with the server.…”