Human Perception of Audio Deepfakes

Müller, Nicolas M.; Markert, Karla; Böttinger, Konstantin

doi:10.1145/3552466.3556531

Cited by 30 publications

(18 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The histogram shows performance is relatively stable across the questions. This observation indicates participants do not improve throughout the task unless they have explicit feedback, as examined by Groh et al (2022) and Müller et al (2022). We quantitatively verified the result by conducting a chi-squared hypothesis test against the uniform distribution, which was not statistically significant (χ 2 = 6.19, p = .99).…”

Section: Participants Do Not Get Better Throughout the Task Without E...mentioning

confidence: 68%

“…Similarly to Groh et al (2022), they found that feedback from the ML model improved human performance. In their experiment, Müller et al (2022) found that the difference between human and AI accuracy was about 10%. However, their study only used English-language clips, only presented one audio clip to participants at a time, and did not collect information about participant confidence.…”

Section: Related Work On Deepfake Detectionmentioning

confidence: 99%

“…However, the experiment is limited to the speaker verification setting. Müller et al (2022) ran a game where 378 participants competed against an ML model to decide if an audio clip was fake. Similarly to Groh et al (2022), they found that feedback from the ML model improved human performance.…”

Section: Related Work On Deepfake Detectionmentioning

confidence: 99%

“…We recruited participants via the Prolific platform. We filtered for participants fluent in English and Mandarin, as fluency affects detection performance (Müller et al, 2022). We paid participants at a rate of £7.25 per hour.…”

Section: Participantsmentioning

confidence: 99%

See 3 more Smart Citations

Warning: Humans Cannot Reliably Detect Speech Deepfakes

Mai¹,

Bray²,

Davies³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

Section: Participants Do Not Get Better Throughout the Task Without E...mentioning

confidence: 68%

Section: Related Work On Deepfake Detectionmentioning

confidence: 99%

Section: Related Work On Deepfake Detectionmentioning

confidence: 99%

Section: Participantsmentioning

confidence: 99%

See 2 more Smart Citations

Warning: Humans Cannot Reliably Detect Speech Deepfakes

Mai¹,

Bray²,

Davies³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…By contrast, our present studies reveal that, in at least some cases, subjects can not only identify that adversarial clips contain speech, but can also comprehend the content of some adversarial audio attacks 9 . Other work has investigated human versus machine perception of “deepfake” stimuli, including both auditory and visual examples (Groh, Epstein, Firestone, & Picard, 2022; Müller, Markert, & Böttinger, 2021).…”

Section: Discussionmentioning

confidence: 99%

Can You Hear Me Now? Sensitive Comparisons of Human and Machine Perception

Lepori¹,

Firestone²

2022

Cognitive Science

View full text Add to dashboard Cite

The rise of machine‐learning systems that process sensory input has brought with it a rise in comparisons between human and machine perception. But such comparisons face a challenge: Whereas machine perception of some stimulus can often be probed through direct and explicit measures, much of human perceptual knowledge is latent, incomplete, or unavailable for explicit report. Here, we explore how this asymmetry can cause such comparisons to misestimate the overlap in human and machine perception. As a case study, we consider human perception of adversarial speech — synthetic audio commands that are recognized as valid messages by automated speech‐recognition systems but that human listeners reportedly hear as meaningless noise. In five experiments, we adapt task designs from the human psychophysics literature to show that even when subjects cannot freely transcribe such speech commands (the previous benchmark for human understanding), they can sometimes demonstrate other forms of understanding, including discriminating adversarial speech from closely matched nonspeech (Experiments 1 and 2), finishing common phrases begun in adversarial speech (Experiments 3 and 4), and solving simple math problems posed in adversarial speech (Experiment 5) — even for stimuli previously described as unintelligible to human listeners. We recommend the adoption of such “sensitive tests” when comparing human and machine perception, and we discuss the broader consequences of such approaches for assessing the overlap between systems.

show abstract