Deep Self-Supervised Hierarchical Clustering for Speaker Diarization

Abstract

Automatic speaker diarization techniques typically involve a two-stage processing approach where audio segments of fixed duration are converted to vector representations in the first stage. This is followed by an unsupervised clustering of the representations in the second stage. In most of the prior approaches, these two stages are performed in an isolated manner with independent optimization steps. In this paper, we propose a representation learning and clustering algorithm that can be iteratively performed for improved speaker diarization. The representation learning is based on principles of selfsupervised learning while the clustering algorithm is a graph structural method based on path integral clustering (PIC). The representation learning step uses the cluster targets from PIC and the clustering step is performed on embeddings learned from the self-supervised deep model. This iterative approach is referred to as self-supervised clustering (SSC). The diarization experiments are performed on CALLHOME and AMI meeting datasets. In these experiments, we show that the SSC algorithm improves significantly over the baseline system (relative improvements of 13% and 59% on CALLHOME and AMI datasets respectively in terms of diarization error rate (DER)). In addition, the DER results reported in this work improve over several other recent approaches for speaker diarization.

Self-Supervised Metric Learning With Graph Clustering For Speaker Diarization

“…This paper extends our previous works on self-supervised learning and graph-based clustering [15,16]. The previous works proposed representation learning and graph based clustering in an iterative self-supervised learning framework.…”

“…In particular, both the embeddings and the adjacency matrix for graph based clustering are jointly learned. Using this joint learning, we show significant performance improvements over baseline systems and previous models based on self-supervised graph clustering methods [15,16].…”

“…The self-supervision in diarization can provide effective representations for downstream tasks without requiring the ground-truth labels [14]. In this paper, we extend our previous work, based on path integral clustering (PIC) [15,16], with metric learning framework inspired by neural PLDA [17]. The previous work explored graph based clustering with self-supervised representation learning.…”

A hybrid HXPLS‐TMFCC parameterization and DCNN‐SFO clustering based speaker diarization system

“…This section describes the performance analysis of the proposed SDS‐HXLP‐DCNN‐SOA speaker diarization scheme depending on metrics, namely, tracking distance, FAR, DER. The performance is likened to the existing methods, such as real‐time implementation of speaker diarization system on Raspberry PI3 utilizing TLBO clustering algorithm (SDS‐RPi3‐TLBO), 35 deep self‐supervised hierarchical clustering for speaker diarization (SDS‐TDNN), 36 meta‐learning with latent space clustering in generative adversarial network for speaker diarization (SDS‐MCGAN), 37 A new method for speaker diarization system utilizing TMFCC parameterization with lion optimization (SDS‐TMFCC‐DNN‐LOA), 38 speaker diarization system utilizing HXLPS with deep neural network (SDS‐HXLP‐DNN), 39 respectively.…”

Chronological Self-Training for Real-Time Speaker Diarization

“…Higher-performing offline algorithms such as k-means or spectral clustering can be used if the clustering is performed after all the points have been collected. Singh and Ganapathy [11] combined representation learning with agglomerative hierarchical clustering (AHC) for significant improvement over AHC alone. A fully-neural offline diarization system can also use unsupervised clustering [12].…”