Assessing Speaker Engagement in 2-Person Debates: Overlap Detection in United States Presidential Debates

Abstract

Co-channel speech recordings typically contain significant amounts of overlap in which the intelligibility and quality of the desired speech is degraded by interference from a competing talker. Convolutive Non-negative Matrix Factorization (CNMF) has been shown to be a successful approach in detecting overlap by extracting specific acoustic basis dimensions for each speaker from an audio stream. While the results of CNMF have been successful, it requires isolated single speech recordings for each speaker to derive their corresponding bases functions/dimensions. In our previous work, The Teager-Kaiser Energy Operator (TEO)-based Pyknogram has been introduced which does not require prior information concerning the speakers. In this study, Pyknogram and CNMF based solutions for overlap detection within audio streams have been examined using the GRID dataset. TEO-based Pyknogram is shown to achieve a relative 8-10% lower Equal Error Rate (EER) compared to CNMF features. Another drawback of CNMF is that its performance drops considerably when dealing with spontaneous speech that has not been considered for extracting bases in the training step. In addition to the experiments on GRID corpus, a secondary evaluation is also performed based on naturalistic audio streams with overlap. Specifically, we collected a real-world audio database of US Presidential debates stemming from the last 12 years that are challenging due to overlap, changing Signal to Interference Ratio (SIR), and environmental noise, etc. Our experiments indicate that TEO-based Pyknogram is well suited for detecting overlap in challenging real world scenarios such as the US presidential debates.

Probabilistic Permutation Invariant Training for Speech Separation

“…The GRID is a multi-speaker, sentence corpus [32], which has been used in monaural speech separation and recognition challenge [33]. Additionally, this corpus has been widely used for assessing the perception of simultaneous speech signals [34,35,36]. This corpus consists of 34 subjects (18 male and 16 female speakers), each narrating 1000 sentences.…”

Single-channel speech separation using Soft-minimum Permutation Invariant Training

“…We generate overlapping speech utterances based on the GRID corpus, which is a multi-speaker, sentence-based corpus used in a monaural speech separation and recognition challenge [42]. This corpus contains 34 speakers, 16 female and 18 male speakers, each providing 1000 sentences, which have been frequently used in several overlapping speech detection and separation studies [43], [44], [45], [37]. To generate mixed speech utterances, random speech recordings are selected from random speakers.…”

Frame-Based Overlapping Speech Detection Using Convolutional Neural Networks

“…In this work we use a multi-speaker, sentence-based corpus called GRID, which has been used in monaural speech separation and recognition challenge [18]. Also, this dataset has been used in several studies [12,19] for overlapping speech detection and separation. This corpus contains 34 speakers, 16 female and 18 male speakers, each narrating 1000 sentence.…”