Privacy Enhanced Multimodal Neural Representations for Emotion Recognition

Jaiswal, Mimansa; Provost, Emily Mower

doi:10.1609/aaai.v34i05.6307

Cited by 38 publications

(9 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…AUC scores for ethnicity are important only for video and, to a lesser extent, multimodal models, meaning that some information is retrievable. Finally, contrary to [23], our multimodal model does not show a higher leakage than our unimodal models.…”

Section: B Experiments Resultscontrasting

confidence: 80%

“…Interestingly, these methodologies are also linked to privacy methods where network designers try to protect their system from attackers trying to retrieve personal information from latent representations. Thus, adversarial learning has been used in the context of speech processing [22], deep visual recognition [21], or multimodal (prosodic and verbal content) emotion recognition [23]. However these approaches always rely on the explicit usage of the protected variables.…”

Section: Fairness Via Adversarial Methodsmentioning

confidence: 99%

“…In the case of a multi-class problem, we report the macro one-vs-rest AUC. This protocol is widely used in the privacy and fairness literature [20], [23]. Good performance of the diagnostic classifiers means that sensitive information is still contained in the latent representation.…”

Section: A Evaluation Metrics and Baseline Classifiersmentioning

confidence: 99%

See 2 more Smart Citations

Don't Judge Me by My Face : An Indirect Adversarial Approach to Remove Sensitive Information From Multimodal Neural Representation in Asynchronous Job Video Interviews

Hemamou¹,

Guillon²,

Martin³

et al. 2021

Preprint

View full text Add to dashboard Cite

Use of machine learning for automatic analysis of job interview videos has recently seen increased interest. Despite claims of fair output regarding sensitive information such as gender or ethnicity of the candidates, the current approaches rarely provide proof of unbiased decision-making, or that sensitive information is not used. Recently, adversarial methods have been proved to effectively remove sensitive information from the latent representation of neural networks. However, these methods rely on the use of explicitly labeled protected variables (e.g. gender), which cannot be collected in the context of recruiting in some countries (e.g. France). In this article, we propose a new adversarial approach to remove sensitive information from the latent representation of neural networks without the need to collect any sensitive variable. Using only a few frames of the interview, we train our model to not be able to find the face of the candidate related to the job interview in the inner layers of the model. This, in turn, allows us to remove relevant private information from these layers. Comparing our approach to a standard baseline on a public dataset with gender and ethnicity annotations, we show that it effectively removes sensitive information from the main network. Moreover, to the best of our knowledge, this is the first application of adversarial techniques for obtaining a multimodal fair representation in the context of video job interviews. In summary, our contributions aim at improving fairness of the upcoming automatic systems processing videos of job interviews for equality in job selection.

show abstract

Section: B Experiments Resultscontrasting

confidence: 80%

Section: Fairness Via Adversarial Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Don't Judge Me by My Face : An Indirect Adversarial Approach to Remove Sensitive Information From Multimodal Neural Representation in Asynchronous Job Video Interviews

Hemamou¹,

Guillon²,

Martin³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In the future, we aim to build our SER model using more complex model structures, e.g., RNN+classifer. We also wish to apply the defense mechanism, such as adversarial training shown in [41], to train the SER model in the FL set up. Finally, we wish to evaluate the membership inference attack within similar experimental settings.…”

Section: Discussionmentioning

confidence: 99%

Attribute Inference Attack of Speech Emotion Recognition in Federated Learning Settings

Feng¹,

Hashemi²,

Hebbar³

et al. 2021

Preprint

View full text Add to dashboard Cite

Speech emotion recognition (SER) processes speech signals to detect and characterize expressed perceived emotions. Many SER application systems often acquire and transmit speech data collected at the client-side to remote cloud platforms for inference and decision making. However, speech data carry rich information not only about emotions conveyed in vocal expressions, but also other sensitive demographic traits such as gender, age and language background. Consequently, it is desirable for SER systems to have the ability to classify emotion constructs while preventing unintended/improper inferences of sensitive and demographic information. Federated learning (FL) is a distributed machine learning paradigm that coordinates clients to train a model collaboratively without sharing their local data. This training approach appears secure and can improve privacy for SER. However, recent works have demonstrated that FL approaches are still vulnerable to various privacy attacks like reconstruction attacks and membership inference attacks. Although most of these have focused on computer vision applications, such information leakages exist in the SER systems trained using the FL technique. To assess the information leakage of SER systems trained using FL, we propose an attribute inference attack framework that infers sensitive attribute information of the clients from shared gradients or model parameters, corresponding to the FedSGD and the FedAvg training algorithms, respectively. As a use case, we empirically evaluate our approach for predicting the client's gender information using three SER benchmark datasets: IEMOCAP, CREMA-D, and MSP-Improv. We show that the attribute inference attack is achievable for SER systems trained using FL. We further identify that most information leakage possibly comes from the first layer in the SER model.

show abstract

“…In [61] Nautsch et al investigate the importance of the development of privacy-preserving technologies to protect speech signals and highlight the importance of applying these technologies to protect speakers and speech characterization in recordings. Some recent works have sought to protect speaker identity [67], gender identity [33] and emotion [2]. VoiceMask, for example, was proposed to mitigate the security and privacy risks of voice input on mobile devices by concealing voiceprints [67].…”

Section: Related Workmentioning

confidence: 99%

Privacy-preserving Voice Analysis via Disentangled Representations

Aloufi

Haddadi

Boyle

2020

Proceedings of the 2020 ACM SIGSAC Conference on Cloud Computing Security Workshop

View full text Add to dashboard Cite

Voice User Interfaces (VUIs) are increasingly popular and built into smartphones, home assistants, and Internet of Things (IoT) devices. Despite offering an always-on convenient user experience, VUIs raise new security and privacy concerns for their users. In this paper, we focus on attribute inference attacks in the speech domain, demonstrating the potential for an attacker to accurately infer a target user's sensitive and private attributes (e.g. their emotion, sex, or health status) from deep acoustic models. To defend against this class of attacks, we design, implement, and evaluate a user-configurable, privacy-aware framework for optimizing speechrelated data sharing mechanisms. Our objective is to enable primary tasks such as speech recognition and user identification, while removing sensitive attributes in the raw speech data before sharing it with a cloud service provider. We leverage disentangled representation learning to explicitly learn independent factors in the raw data. Based on a user's preferences, a supervision signal informs the filtering out of invariant factors while retaining the factors reflected in the selected preference. Our experimental evaluation over five datasets shows that the proposed framework can effectively defend against attribute inference attacks by reducing their success rates to approximately that of guessing at random, while maintaining accuracy in excess of 99% for the tasks of interest. We conclude that negotiable privacy settings enabled by disentangled representations can bring new opportunities for privacy-preserving applications.

show abstract

Privacy Enhanced Multimodal Neural Representations for Emotion Recognition

Cited by 38 publications

References 20 publications

Don't Judge Me by My Face : An Indirect Adversarial Approach to Remove Sensitive Information From Multimodal Neural Representation in Asynchronous Job Video Interviews

Don't Judge Me by My Face : An Indirect Adversarial Approach to Remove Sensitive Information From Multimodal Neural Representation in Asynchronous Job Video Interviews

Attribute Inference Attack of Speech Emotion Recognition in Federated Learning Settings

Privacy-preserving Voice Analysis via Disentangled Representations

Contact Info

Product

Resources

About