Speaker trait characterization in web videos: Uniting speech, language, and facial features

Abstract

We present a multi-modal approach to speaker characterization using acoustic, visual and linguistic features. Full realism is provided by evaluation on a database of real-life web videos and automatic feature extraction including face and eye detection, and automatic speech recognition. Different segmentations are evaluated for the audio and video streams, and the statistical relevance of Linguistic Inquiry and Word Count (LIWC) features is confirmed. In the result, late multimodal fusion delivers 73, 92 and 73 % average recall in binary age, gender and race classification on unseen test subjects, outperforming the best single modalities for age and race.

Recent developments in openSMILE, the munich open-source multimedia feature extractor

“…Let us now turn to multimedia feature extraction by considering speaker characterization in web videos as in [10]. An audio-visual feature extraction scheme similar to the one described in [10] using a proprietary implementation of video feature extraction can now be realized by using openSMILE exclusively.…”

“…An audio-visual feature extraction scheme similar to the one described in [10] using a proprietary implementation of video feature extraction can now be realized by using openSMILE exclusively. The underlying idea is close to the 'toy' example from Figure 1, yet extracting over 1.5 k LLD-functional combinations from the audio (INTERSPEECH 2010 set, IS10, delivered as configuration file with the openSMILE distribution) and using sliding window lengths of 4 s. The configuration file for synchronized audio and video feature extraction is also delivered with the current release candidate.…”

“…Table 4), 96.4 % and 70.0 % recall are obtained for gender and ethnicity, which is notable given the real-life nature of the task. Details on the classification procedure using Weka's [11] SVM can be found in [10].…”