Personalized Federated Learning with Attention Mechanisms

Abstract

With the rapid development of science and technology, machine learning and deep learning technology have more and more applications in our daily life. At the same time, people pay more and more attention to the protection of their own privacy. And in recent years, various countries have introduced a series of laws and regulations to protect people's data privacy. In this case, the traditional method of pooling the data of each user for model training is no longer applicable, and the data of each user can only be saved in the hands of the user himself, which has a profound impact on the development of artificial intelligence technology. To solve this dilemma, the industry has proposed the concept of federated learning. Due to the data imbalance problem between each user, the prediction ability of the global aggregation model obtained by federated learning on the local specific data of each user needs to be improved. Therefore, personalization is an issue that federated learning needs to pay attention to.In order to improve the personalization ability, this paper proposes a federated transfer learning algorithm with attention mechanism. After each user obtains the global aggregation model of federated learning, the attention module is added to the local model, and then the parameters of the low-level neurons are frozen during training, and only the parameters of the high-level neurons and the attention module are updated. Finally, each user obtains a unique model that is more suitable for local data. In this paper, we conduct experiments to analyze the performance of this algorithm and the federated transfer algorithm and federated learning algorithm in terms of accuracy. The experimental results show that on the convolutional neural network model, the federated transfer learning algorithm applying the attention mechanism has improved the accuracy compared with the federated transfer algorithm and the federated learning algorithm.