Privacy Enhanced Multimodal Neural Representations for Emotion Recognition

Jaiswal, Mimansa; Provost, Emily Mower

doi:10.48550/arxiv.1910.13212

Cited by 4 publications

(5 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most of the proposed works focus on protecting/anonymizing speaker identity using voice conversion (VC) mechanisms [11,81,94,99]. Beyond speaker identity, various works propose to protect speaker gender [49] and emotion [13], wherein an edge-based system is proposed to filter affect patterns from a user's voice before sharing it with cloud services for further analysis. Another direction is protecting users' privacy by ensuring that sensitive data is not unnecessarily transmitted to service providers [15].…”

Section: Privacy Guardmentioning

confidence: 99%

See 1 more Smart Citation

Locally Authenticated Privacy-preserving Voice Input

Aloufi¹,

Nautsch²,

Haddadi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Increasing use of our biometrics (e.g., fingerprints, faces, or voices) to unlock access to and interact with online services raises concerns about the trade-offs between convenience, privacy, and security. Service providers must authenticate their users, although individuals may wish to maintain privacy and limit the disclosure of sensitive attributes beyond the authentication step, e.g., when interacting with Voice User Interfaces (VUIs). Preserving privacy while performing authentication is challenging, particularly where adversaries can use biometric data to train transformation tools (e.g., 'deepfaked' speech) and use the faked output to defeat existing authentication systems. In this paper, we take a step towards understanding security and privacy requirements to establish the threat and defense boundaries. We introduce a secure, flexible privacypreserving system to capture and store an on-device fingerprint of the users' raw signals (i.e., voice) for authentication instead of sending/sharing the raw biometric signals. We then analyze this fingerprint using different predictors, each evaluating its legitimacy from a different perspective (e.g., target identity claim, spoofing attempt, and liveness). We fuse multiple predictors' decisions to make a final decision on whether the user input is legitimate or not. Validating legitimate users yields an accuracy rate of 98.68% after cross-validation using our verification technique. The pipeline runs in tens of milliseconds when tested on a CPU and a single-core ARM processor, without specialized hardware.

show abstract

Section: Privacy Guardmentioning

confidence: 99%

“…The word error rate (WER; lower is better) and realtime factor (RTF; lower RTF is more computationally efficient). 49. 18.45 7.35 7.14 2.11 5.95 1.33 5.36 2.20 Private Data 23.21 51.41 85.00 2.87 32.74 1.39 27.38 1.04 11.31 2.06…”

mentioning

confidence: 99%

Locally Authenticated Privacy-preserving Voice Input

Aloufi¹,

Nautsch²,

Haddadi³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…It has been found that when the attacker has complete knowledge of the VC scheme and target speaker mapping, none of the existing VC methods can protect speaker identity. Beyond speaker identity, various works propose to protect speaker gender [12] and emotion [13], wherein an edge-based system is proposed to filter affect patterns from a user's voice before sharing it with cloud services for further analysis.…”

Section: Privacy-preserving Voice Analyticsmentioning

confidence: 99%

Configurable Privacy-Preserving Automatic Speech Recognition

Aloufi¹,

Haddadi²,

Boyle³

2021

Preprint

View full text Add to dashboard Cite

Voice assistive technologies have given rise to far-reaching privacy and security concerns. In this paper we investigate whether modular automatic speech recognition (ASR) can improve privacy in voice assistive systems by combining independently trained separation, recognition, and discretization modules to design configurable privacy-preserving ASR systems. We evaluate privacy concerns and the effects of applying various stateof-the-art techniques at each stage of the system, and report results using task-specific metrics (i.e. WER, ABX, and accuracy). We show that overlapping speech inputs to ASR systems present further privacy concerns, and how these may be mitigated using speech separation and optimization techniques. Our discretization module is shown to minimize paralinguistics privacy leakage from ASR acoustic models to levels commensurate with random guessing. We show that voice privacy can be configurable, and argue this presents new opportunities for privacy-preserving applications incorporating ASR.

show abstract

“…Privacy preserved representation learning is a relatively unexplored research topic. Recently, researchers have started to utilise privacy-preserving representation learning models to protect speaker identity [383], gender identity [384]. To preserve users' privacy, federated learning [385] is another alternative setting where the training of a shared global model is performed using multiple participating computing devices.…”

Section: F Privacy Preserving Representationsmentioning

confidence: 99%

Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends

Latif,

Rana,

Khalifa

et al. 2020

Preprint

View full text Add to dashboard Cite

Research on speech processing has traditionally considered the task of designing hand-engineered acoustic features (feature engineering) as a separate distinct problem from the task of designing efficient machine learning (ML) models to make prediction and classification decisions. There are two main drawbacks to this approach: firstly, the feature engineering being manual is cumbersome and requires human knowledge; and secondly, the designed features might not be best for the objective at hand. This has motivated the adoption of a recent trend in speech community towards utilisation of representation learning techniques, which can learn an intermediate representation of the input signal automatically that better suits the task at hand and hence lead to improved performance. The significance of representation learning has increased with advances in deep learning (DL), where the representations are more useful and less dependent on human knowledge, making it very conducive for tasks like classification, prediction, etc. The main contribution of this paper is to present an up-to-date and comprehensive survey on different techniques of speech representation learning by bringing together the scattered research across three distinct research areas including Automatic Speech Recognition (ASR), Speaker Recognition (SR), and Speaker Emotion Recognition (SER). Recent reviews in speech have been conducted for ASR, SR, and SER, however, none of these has focused on the representation learning from speech-a gap that our survey aims to bridge. We also highlight different challenges and the key characteristics of representation learning models and discuss important recent advancements and point out future trends. Our review can be used by the speech research community as an essential resource to quickly grasp the current progress in representation learning and can also act as a guide for navigating study in this area of research.

show abstract

Privacy Enhanced Multimodal Neural Representations for Emotion Recognition

Cited by 4 publications

References 15 publications

Locally Authenticated Privacy-preserving Voice Input

Locally Authenticated Privacy-preserving Voice Input

Configurable Privacy-Preserving Automatic Speech Recognition

Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends

Contact Info

Product

Resources

About