Who is Real Bob? Adversarial Attacks on Speaker Recognition Systems

Abstract

Speaker recognition (SR) is widely used in our daily life as a biometric authentication or identification mechanism. The popularity of SR brings in serious security concerns, as demonstrated by recent adversarial attacks. However, the impacts of such threats in the practical black-box setting are still open, since current attacks consider the white-box setting only.In this paper, we conduct the first comprehensive and systematic study of the adversarial attacks on SR systems (SRSs) to understand their security weakness in the practical blackbox setting. For this purpose, we propose an adversarial attack, named FAKEBOB, to craft adversarial samples. Specifically, we formulate the adversarial sample generation as an optimization problem, incorporated with the confidence of adversarial samples and maximal distortion to balance between the strength and imperceptibility of adversarial voices. One key contribution is to propose a novel algorithm to estimate the score threshold, a feature in SRSs, and use it in the optimization problem to solve the optimization problem. We demonstrate that FAKEBOB achieves close to 100% targeted attack success rate on both open-source and commercial systems. We further demonstrate that FAKEBOB is also effective (at least 65% untargeted success rate) on both open-source and commercial systems when playing over the air in the physical world. Moreover, we have conducted a human study which reveals that it is hard for human to differentiate the speakers of the original and adversarial voices. Last but not least, we show that three promising defense methods for adversarial attack from the speech recognition domain become ineffective on SRSs against FAKEBOB, which calls for more effective defense methods. We highlight that our study peeks into the security implications of adversarial attacks on SRSs, and realistically fosters to improve the security robustness of SRSs.

On the Detection of Adaptive Adversarial Attacks in Speaker Verification Systems

“…As shown in Section IV, one straightforward countermeasure by attackers is to reduce the perturbation threshold so that the standard deviation of perturbations (i.e., σ) can be less than σ D in Equation (23). However, as shown in [3] and in our experiments in Section VI-B, when decreases, the attack success rate of an adversarial attack would decrease as well. That is, a small value of can let adversarial attacks reduce the attacking power in the first place.…”

“…An attacker can design an adversarial example attack to make the SV system falsely accept an illegal user as a legitimate user. FAKEBOB is the state-of-the-art black-box adversarial attack against popular score-based SV systems such as GMM and i-vector [3]. The basic idea of FAKEBOB attacks is to find small perturbations p[n], so that an SV system would reject s should be small.…”

“…The implementation of the FAKEBOB attack is summarized in Algorithm 1. In [3], FAKEBOB is shown to be able to achieve a very high targeted attack success rate on popular SV systems.…”

“…Such a sniffing and spoofing attack requires an attacker to obtain a legitimate user's audio. The other attack is called adversarial attacks that generate a speech acceptable to the system by adding small and well-designed perturbations to an illegal user's speech [3], [1]. Such an attack does not need a copy of the legitimate user's speech and is imperceptible to humans.…”

“…Since the currently widely-used speaker verification systems, such as GMM, ivector, d-vector, and x-vector, are based on machine learning, they are vulnerable to adversarial attacks. Specially, Chen et al designed a black-box adversarial attack, called FAKEBOB, that does not require the implementation information of a speaker verification system and is effective against both open-source and commercial systems [3]. Moreover, it has been shown in [3] that several defense methods, including local smoothing, quantization, and temporal dependency detection [21], that work well against adversarial attacks in the image domain are not able to counteract FAKEBOB.…”