Weakly Supervised Training of Hierarchical Attention Networks for Speaker Identification

Abstract

Identifying multiple speakers without knowing where a speaker's voice is in a recording is a challenging task. In this paper, a hierarchical attention network is proposed to solve a weakly labelled speaker identification problem. The use of a hierarchical structure, consisting of a frame-level encoder and a segment-level encoder, aims to learn speaker related information locally and globally. Speech streams are segmented into fragments. The frame-level encoder with attention learns features and highlights the target related frames locally, and output a fragment based embedding. The segment-level encoder works with a second attention layer to emphasize the fragments probably related to target speakers. The global information is finally collected from segment-level module to predict speakers via a classifier. To evaluate the effectiveness of the proposed approach, artificial datasets based on Switchboard Cellular part1 (SWBC) and Voxceleb1 are constructed in two conditions, where speakers' voices are overlapped and not overlapped. Comparing to two baselines the obtained results show that the proposed approach can achieve better performances. Moreover, further experiments are conducted to evaluate the impact of utterance segmentation. The results show that a reasonable segmentation can slightly improve identification performances.

T-vectors: Weakly Supervised Speaker Identification Using Hierarchical Transformer Model

“…The final speaker identities are the output vector which contains the scores (between 1 and 0) for each speaker. The model is trained using binary cross entropy loss (shown in Eq 1) [7,6]:…”

“…For this work, as there is no data for weakly supervised speaker identification task, and to compare with our previous published results, the data construction process is the same as that in our previous work [6].…”

“…This kind of weakly labelled large data collections are available online [3]. Making use of such datasets might be beneficial for training with a large amount of data [6].…”

“…Our previous work [6] proposed a hierarchical attention network (H-vectors) to solve the weakly supervised speaker identification problem. However, the H-vectors proposed in [6] might have two disadvantages: the use of the GRU in each frame-level encoder might influence the training speed and cause gradient vanishing problem, and there is no connections between each frame-level encoders, which might influence the performance.…”

“…In this work, a hierarchical network based on transformer's encoder architecture [9] is proposed to tackle the weakly supervised speaker identification problem. Based on our previous work [6], the proposed model (T-vector) contains a frame-level encoder and segment-level encoder. However, the T-vector model makes use of transformer encoder blocks in both frame-level encoder and segment-level encoder to better capture speaker information and process the sequence in parallel instead using GRU layer [10].…”