Tandem Assessment of Spoofing Countermeasures and Automatic Speaker Verification: Fundamentals

Abstract

Recent years have seen growing efforts to develop spoofing countermeasures (CMs) to protect automatic speaker verification (ASV) systems from being deceived by manipulated or artificial inputs. The reliability of spoofing CMs is typically gauged using the equal error rate (EER) metric. The primitive EER fails to reflect application requirements and the impact of spoofing and CMs upon ASV and its use as a primary metric in traditional ASV research has long been abandoned in favour of risk-based approaches to assessment. This paper presents several new extensions to the tandem detection cost function (t-DCF), a recent risk-based approach to assess the reliability of spoofing CMs deployed in tandem with an ASV system. Extensions include a simplified version of the t-DCF with fewer parameters, an analysis of a special case for a fixed ASV system, simulations which give original insights into its interpretation and new analyses using the ASVspoof 2019 database. It is hoped that adoption of the t-DCF for the CM assessment will help to foster closer collaboration between the anti-spoofing and ASV research communities.

ASVspoof 2019: Spoofing Countermeasures for the Detection of Synthesized, Converted and Replayed Speech

“…While the parameter-free equal error rate (EER) metric is retained as a secondary metric, the primary metric is the tandem detection cost function (t-DCF) [5], and the specific ASVconstrained variant detailed in [6]. The detection threshold (set to the EER operating point) of the ASV system (designed by the organiser) is fixed, whereas the detection threshold of the CM system (designed by participants) is allowed to vary.…”

“…Whereas previous editions of ASVspoof utilised the equal error rate (EER) metric to judge performance, the 2019 edition shifted to the ASV-centric tandem detection cost function (t-DCF) metric [5], [6]. While the latter reflects the impact of both spoofing and countermeasures…”

On Joint Optimization of Automatic Speaker Verification and Anti-Spoofing in the Embedding Space

“…Just like ASV, the EER metric is typically used to evaluate standalone anti-spoofing systems, where false rejection happens when a bonafide speech utterance is detected as a spoofing attack, and false acceptance occurs when spoofed speech is detected as bonafide speech. Recently, the ASV-constrained minimum tandem detection cost function (min-tDCF) metric [41] was proposed to evaluate a PAD system given a fixed ASV system, considering the priors and costs of the different hypotheses. This was the primary metric used in the last ASVspoof 2019 challenge [13].…”

“…However, the EER does not account for the costs of missing target users and falsely accepting zero-effort impostors or spoofing attacks, nor the prior probabilities of each. To take these costs and priors into account, the min-tDCF [41], [52] has been recently proposed as a metric for evaluating decision-level integration systems. Nevertheless, decision-level integration systems assume that there are two separate systems (ASV and PAD) with two different operating thresholds which make their own binary decisions independently.…”

“…Likewise, the evaluation of the standalone antispoofing systems was done in terms of pooled EER. Once we had the ASV and PAD scores, we also evaluated the ASV-constrained min-tDCF [41] for both the LA and PA scenarios, separately. These metrics (EER and min-tDCF) have been evaluated using the optimal threshold for each metric, as the ASVspoof 2019 challenge did for evaluating every antispoofing system.…”