Characterizing the Effect of Audio Degradation on Privacy Perception And Inference Performance in Audio-Based Human Activity Recognition

Abstract

Audio has been increasingly adopted as a sensing modality in a variety of human-centered mobile applications and in smart assistants in the home. Although acoustic features can capture complex semantic information about human activities and context, continuous audio recording often poses significant privacy concerns. An intuitive way to reduce privacy concerns is to degrade audio quality such that speech and other relevant acoustic markers become unintelligible, but this often comes at the cost of activity recognition performance. In this paper, we employ a mixed-methods approach to characterize this balance. We first conduct an online survey with 266 participants to capture their perception of privacy qualitatively and quantitatively with degraded audio. Given our findings that privacy concerns can be significantly reduced at high levels of audio degradation, we then investigate how intentional degradation of audio frames can affect the recognition results of the target classes while maintaining effective privacy mitigation. Our results indicate that degradation of audio frames can leave minimal effects for audio recognition using frame-level features. Furthermore, degradation of audio frames can hurt the performance to some extend for audio recognition using segment-level features, though the usage of such features may still yield superior recognition performance. Given the different requirements on privacy mitigation and recognition performance for different sensing purposes, such trade-offs need to be balanced in actual implementations. CCS CONCEPTS • Security and privacy → Domain-specific security and privacy architectures; Privacy protections; • Human-centered computing → Empirical studies in ubiquitous and mobile computing.

Safe audio AI services in smart buildings

“…[22]. Expanding on previous work in the area of audio scrambling [13], we suggest increased research to explore adversarial AI components that check whether speech content and individual identity could be reconstructed from degraded or scrambled audio. The purpose of the adversarial component (which we have defined as system A) is to enhance the trust partnership between AI and consumers.…”

“…Many people have a reasonable expectation of privacy when it comes to how their smart technologies store, process, and transmit audio data [5]. Recent work in audio-based activity recognition has shown that people begin to trust audio privacy more if monitored speech has been rendered unintelligible to humans through an audio degradation technique called scrambling [13]. Some types of scrambling or noise may make audio content unintelligible to humans, but does not neutralise information in the audio signal for an AI system.…”

Theophany

“…However, this solution is primarily designed for text data and is not directly applicable to preserve privacy for daily conversations. Liang et al [18] study the users' perception in an environment in which the audio is recorded continuously. The authors propose that users' privacy perception improves if the quality of the audio is degraded before it is recorded.…”

“…Speech augmentation threatens this paradigm by recording and transmitting information over digital means. Yet, few studies consider privacy in face-to-face conversations, and most of the proposed solutions reduce speech intelligibility [4,10,18,20,29].…”

Transferring Voice Knowledge for Acoustic Event Detection: An Empirical Study

“…Transferring knowledge from a source domain to a target task can be a useful way to enrich the learning of the target dataset [7]. It is particularly meaningful in real-world audio analysis where the target audio accessibility can be limited due to challenges such as scalability [17] and privacy constraints [18]. For acoustic classification, transfer learning has been successfully applied both across tasks [19,20,21,22,23] and across modalities [5,24].…”