Privacy-preserving Speech Emotion Recognition through Semi-Supervised Federated Learning

Abstract

Speech Emotion Recognition (SER) refers to the recognition of human emotions from natural speech. If done accurately, it can offer a number of benefits in building humancentered context-aware intelligent systems. Existing SER approaches are largely centralized, without considering users' privacy. Federated Learning (FL) is a distributed machine learning paradigm dealing with decentralization of privacy-sensitive personal data. In this paper, we present a privacy-preserving and data-efficient SER approach by utilizing the concept of FL. To the best of our knowledge, this is the first federated SER approach, which utilizes self-training learning in conjunction with federated learning to exploit both labeled and unlabeled on-device data. Our experimental evaluations on the IEMOCAP dataset shows that our federated approach can learn generalizable SER models even under low availability of data labels and highly non-i.i.d. distributions. We show that our approach with as few as 10% labeled data, on average, can improve the recognition rate by 8.67% compared to the fully-supervised federated counterparts.

Semi-FedSER: Semi-supervised Learning for Speech Emotion Recognition On Federated Learning using Multiview Pseudo-Labeling

“…However, highquality labeled data samples do not often exist in real-life settings, and most data samples are indeed unlabeled. In order to address the limited labeled data samples in an FL setting, prior works have considered using semi-supervised learning (SSL), which utilizes unlabeled samples in addition to a small portion of labeled examples to obtain desired model performance [7,8]. SemiFed is an SSL framework that integrates supervised and unsupervised training during FL.…”

“…In addition, we propose to use the past pseudo label information to improve the quality of the pseudo labels generated. To the best of our knowledge, the only similar work that exists in the literature is [8] for SER application. We show that our Semi-FedSER can generate desired SER performance even when the local label rate l = 20% and data distribution is completely non-iid using IEMOCAP [13], and MSP-Improv [14] datasets for the experiments.…”