Asynchronous Online Federated Learning for Edge Devices with Non-IID Data

Abstract

Federated learning (FL) is a machine learning paradigm where a shared central model is learned across multiple distributed client devices while the training data remains on edge devices or local clients. Most prior work on federated learning uses Federated Averaging (FedAvg) as an optimization method for training in a synchronized fashion. This involves independent training at multiple edge devices with synchronous aggregation steps. However, the assumptions made by FedAvg are not realistic given the heterogeneity of devices. In particular, the volume and distribution of collected data vary in the training process due to different sampling rates of edge devices. The edge devices themselves also vary in their available communication bandwidth and system configurations, such as memory, processor speed, and power requirements. This leads to vastly different training times as well as model/data transfer times. Furthermore, availability issues at edge devices can lead to a lack of contribution from specific edge devices to the federated model. In this paper, we present an Asynchronous Online Federated Learning (ASOfed) framework, where the edge devices perform online learning with continuous streaming local data and a central server aggregates model parameters from local clients. Our framework updates the central model in an asynchronous manner to tackle the challenges associated with both varying computational loads at heterogeneous edge devices and edge devices that lag behind or dropout. Experiments on three real-world datasets show the effectiveness of ASO-fed on lowering the overall training cost and maintaining good prediction performance.

Federated Action Recognition on Heterogeneous Embedded Devices

“…For federated learning on heterogeneous devices, previous work include an adaptive control algorithm that determines the best trade-off between local updates and global aggregation under a given resource constraint [38], model training in a network of heterogeneous edge devices, taking into account communication costs [39], and a method for straggler acceleration by dynamically masking neurons [40]. Aso-fed [41] presents an online learning algorithm that updates the central model in an asynchronous manner, tackling challenges associated with both varying computational loads at heterogeneous edge devices and stragglers. There has been a sparse but rapidly growing work in federated learning at edge devices, driven by the increasing numbers of such edge devices [42], [43], [38].…”

Towards On-Device Federated Learning: A Direct Acyclic Graph-based Blockchain Approach

“…The authors in [5] demonstrate that asynchronous scheme is an attractive approach to mitigate stragglers in heterogeneous environments. As FL is more likely to be deployed in mobile wireless networks which are usually heterogeneous environment in the future, some researches on designing the asynchronous FL systems have been investigated [30,31]. In asynchronous FL, a node can download a global model from a central server and train a local model to upload in idle state at any time.…”

Implicit model specialization through dag-based decentralized federated learning

“…Another common challenge for federated learning and other decentralized learning approaches is the difference in data distributions present for different clients. [9,24,69] For this not independent and identically distributed (non-IID) data, model updates could counteract each other and hinder the training progress. [69] In this paper, we demonstrate for the first time how the seemingly diverse goals of distributed model training, model personalization as well as robustness against poisoning attacks, can be addressed by a single mechanism that is inspired by distributed ledgers and federated averaging.…”