Federated Virtual Learning on Heterogeneous Data with Local-global Distillation

Abstract

Despite Federated Learning (FL)'s trend for learning machine learning models in a distributed manner, it is susceptible to performance drops when training on heterogeneous data. Recently, dataset distillation has been explored in order to improve the efficiency and scalability of FL by creating a smaller, synthetic dataset that retains the performance of a model trained on the local private datasets. We discover that using distilled local datasets can amplify the heterogeneity issue in FL. To address this, we propose a new method, called Federated Virtual Learning on Heterogeneous Data with Local-Global Distillation (FEDLGD), which trains FL using a smaller synthetic dataset (referred as virtual data) created through a combination of local and global distillation. Specifically, to handle synchronization and class imbalance, we propose iterative distribution matching to allow clients to have the same amount of balanced local virtual data; to harmonize the domain shifts, we use federated gradient matching to distill global virtual data that are shared with clients without hindering data privacy to rectify heterogeneous local training via enforcing local-global feature similarity. We experiment on both benchmark and real-world datasets that contain heterogeneous data from different sources. Our method outperforms state-of-the-art heterogeneous FL algorithms under the setting with a very limited amount of distilled virtual data.

The Construction of City Image of Macau as Represented in the Film “A City Called Macau”

“…(1) Local logits uploading: Clients train their local models using private data and subsequently submit the logits of these models on a public dataset to the central server. Parameter-based [129] Data-based [188] Privacy No need to share parameters, less privacy risk [35] Need to share parameters, which poses privacy risks [31] Need to upload local data (compressed), which lead to privacy concerns Non-IID Clients with heterogeneous data can learn from each other, mitigating non-IID [156] Clients with heterogeneous data can learn from each other, mitigating non-IID [44] Can effectively solve non-IID [51] Communication No need to share parameters, low communication cost [59] Need to share parameters, high communication cost…”