Decentralized Federated Learning Preserves Model and Data Privacy

Abstract

The increasing complexity of IT systems requires solutions, that support operations in case of failure. Therefore, Artificial Intelligence for System Operations (AIOps) is a field of research that is becoming increasingly focused, both in academia and industry. One of the major issues of this area is the lack of access to adequately labeled data, which is majorly due to legal protection regulations or industrial confidentiality. Methods to mitigate this stir from the area of federated learning, whereby no direct access to training data is required. Original approaches utilize a central instance to perform the model synchronization by periodical aggregation of all model parameters. However, there are many scenarios where trained models cannot be published since its either confidential knowledge or training data could be reconstructed from them. Furthermore the central instance needs to be trusted and is a single point of failure. As a solution, we propose a fully decentralized approach, which allows to share knowledge between trained models. Neither original training data nor model parameters need to be transmitted. The concept relies on teacher and student roles that are assigned to the models, whereby students are trained on the output of their teachers via synthetically generated input data. We conduct a case study on log anomaly detection. The results show that an untrained student model, trained on the teachers output reaches comparable F1-scores as the teacher. In addition, we demonstrate that our method allows the synchronization of several models trained on different distinct training data subsets.

Artificial Intelligence for IT Operations (AIOPS) Workshop White Paper

“…Decentralized Federated Learning Preserves Model and Data Privacy Wittkopp and Acker [21] propose a decentralized federated learning approach for sharing knowledge between different IT-services in a privacy-aware procedure. The evaluation shows improvements for log-data anomaly detection when training DeepLog models with a teacher-student approach without sharing directly training data or any model parameters.…”

Distributed Deep Learning in Open Collaborations

“…In Federated Learning, multiple parties train a shared model on decentralized privacy-sensitive data that cannot be shared between devices [59]. For that reason, federated learning algorithms prioritize data privacy over training efficiency, often leaving most of the compute resources unused [60,61]. For a more detailed overview of Federated Learning, refer to Appendix A.…”

“…Maintaining data privacy in these conditions also requires specialized techniques that introduce communication overhead. For instance, [61] proposes a system where workers cannot share parameters directly, relying on a secure peer-to-peer knowledge distillation instead.…”

Training Data Reduction for Performance Models of Data Analytics Jobs in the Cloud

“…The second considerable issue is privacy. While there are attempts at preserving data privacy in federated learning [12], [13], researchers repeatedly find vulnerabilities and manage to reconstruct the original data [14], [15]. These considerations were taken into account in our previous work, C3O, where users have full control over their data as they make contributions to the public training data repository on an entirely voluntary basis.…”