ALBAYZIN 2016 spoken term detection evaluation: an international open competitive evaluation in Spanish

Abstract

Within search-on-speech, Spoken Term Detection (STD) aims to retrieve data from a speech repository given a textual representation of a search term. This paper presents an international open evaluation for search-on-speech based on STD in Spanish and an analysis of the results. The evaluation has been designed carefully so that several analyses of the main results can be carried out. The evaluation consists in retrieving the speech files that contain the search terms, providing their start and end times, and a score value that reflects the confidence given to the detection. Two different Spanish speech databases have been employed in the evaluation: MAVIR database, which comprises a set of talks from workshops, and EPIC database, which comprises a set of European Parliament sessions in Spanish. We present the evaluation itself, both databases, the evaluation metric, the systems submitted to the evaluation, the results, and a detailed discussion. Five different research groups took part in the evaluation, and ten different systems were submitted in total. We compare the systems submitted to the evaluation and make a deep analysis based on some search term properties (term length, within-vocabulary/out-of-vocabulary terms, single-word/ multi-word terms, and native (Spanish)/foreign terms).

Cenatav Voice Group System for Albayzin 2018 Search on Speech Evaluation

“…Comparing with results obtained with the same MAVIR dataset in Albayzin 2016 spoken term detection evaluation [5], our results, for all the proposed models, are similar to obtained by the best system with the DEV set (first line of table 9 of [5]) and the TEST set (first line of table 10 of [5]). Additionally, proposed DNN-HMM model surpasses the behavior of the best system in that evaluation.…”

“…Spoken Term Detection (STD), is defined by NIST [1] as "searching vast, heterogeneous audio archives for occurrences of spoken terms". Iberian institutions have been conducted researches recently on this task, as shown in previous Albayzin Challenges [2,3,4,5].…”

Search on speech from spoken queries: the Multi-domain International ALBAYZIN 2018 Query-by-Example Spoken Term Detection Evaluation

“…ALBAYZIN evaluation campaigns comprise an internationally open set of evaluations supported by the Spanish Thematic Network on Speech Technologies (RTTH 1 ) and the ISCA Special Interest Group on Iberian Languages (SIG-IL 2 ), which have been held biennially since 2006. These evaluation campaigns provide an objective mechanism to compare different systems and are a powerful way to promote research on different speech technologies [78][79][80][81][82][83][84][85][86][87].…”

ALBAYZIN 2018 spoken term detection evaluation: a multi-domain international evaluation in Spanish

“…ALBAYZIN evaluation campaigns comprise an internationally open set of evaluations supported by the Spanish Thematic Network on Speech Technologies (RTTH 1 ) and the ISCA Special Interest Group on Iberian Languages (SIG-IL 2 ), which have been held biennially since 2006. These evaluation campaigns provide an objective mechanism to compare different systems and are a powerful way to promote research on different speech technologies [77][78][79][80][81][82][83][84][85][86].…”