Robust Multichannel Gender Classification from Speech in Movie Audio

Abstract

Speech in the form of scripted dialogues forms an important part of the audio signal in movies. However, it is often masked by background audio signals such as music, ambient noise or background chatter. These background sounds make even otherwise simple tasks, such as gender classification, challenging. Additionally, the variability in this noise across movies renders standard approaches to source separation or enhancement inadequate. Instead, we exploit multichannel information present in different language channels (English, Spanish, French) for each movie to improve the robustness of our gender classification system. We exploit the fact that the speaker labels of interest in this case co-occur in each language channel. We fuse the predictions obtained for each channel using Recognition Output Voting Error Reduction (ROVER) and show that this approach improves the gender accuracy by 7% absolute (11% relative) compared to the best independent prediction on any single channel. In the case of surround movies, we further investigate fusion of mono audio and front center channels which shows 5% and 3% absolute (8% and 4% relative) increase in accuracy compared to only using mono and front center channel, respectively.

Improving Gender Identification in Movie Audio Using Cross-Domain Data

“…We performed a comprehensive evaluation of our system by first evaluating the BLSTM-VAD that was trained with movie audio forced-aligned with the subtitles using Gentle. We compared the performance of our VAD to that of the state-of-the-art for movie data, and also against openSMILE VAD, which is used in [5]. We then evaluated all gender identification models with, a) ground-truth (or oracle) VAD [18], and b) the proposed BLSTM-VAD (see Sec 4).…”

“…The dataset was partitioned into 82 movies for training and 13 movies for system development. The development split was chosen to have a sample representative of different movie genres [5].…”

“…This resulted in 128.5 and 22.4 minutes of speech from male and female speakers re- spectively from a total audio duration of 8hrs. This allowed us to test the performance of our system at a higher precision than the previous methods which evaluated at the subtitle level [5].…”

“…We use the WSJ-SI84 dataset, a subset of the WSJ corpus, in order to replicate the experiments performed in [5]. It consists of a total of 15 hours of data with 7184 utterances from 42 male and 41 female speakers.…”

“…Gender classification from face images in the wild is possible with an accuracy of over 95% [2]. By comparison, gender identification from audio has also been known to show similar performance [3,4], although to the best of our knowledge, an accuracy of 71% has been reported on movie audio [5]. Besides obtaining measures such as speaking time, gender identification can also be an important step for several audio tasks such as speech recognition [6], speaker segmentation [7] and emotion recognition [8].…”