Presentation Matters: Evaluating Speaker Identification Tasks

Abstract

This paper details our evaluations and comparisons of speaker identification (SID) performance by listeners across different tasks. Experiment 1 participants completed traditional targetlineup (1-out-of-N speakers or out-of-set speaker) and binary (speaker verification) tasks. Experiment 2 participants completed trials online by using a clustering method by grouping speech recordings into speaker-specific clusters. Both studies employed similar speech recordings from the PTSVOX corpus. Our results showed participants who completed the binary and clustering tasks had higher accuracy than those who completed the target-lineup task. We also observed that independent of the tasks participants found some speakers significantly more difficult to identify relative to their foils. Pearson correlation procedures showed significant negative correlations between accuracy and task-dependent temporal-based metrics across tasks, where an increase in time required to make determinations yielded a decrease in perceptual SID performance. These findings underscored the important role of SID task design and the process of selecting speech recordings. Future work aims to examine the relationship between different perceptual SID task performances and scores generated by automatic speaker verification systems.

Evaluating the effects of task design on unfamiliar Francophone listener and automatic speaker identification performance

“…By instructing participants to sort 32 stimuli into 1 to 32 different speaker groups, Johnson et al suppressed the possibility of introducing artificial bias [21]. Recently O'Brien et al [44][45][46] developed a perceptual clustering method and reported unfamiliar listeners were effective at navigating the intuitive interface.…”