A neural network approach to audio-assisted movie dialogue detection

Abstract

A novel framework for audio-assisted dialogue detection based on indicator functions and neural networks is investigated. An indicator function defines that an actor is present at a particular time instant. The cross-correlation function of a pair of indicator functions and the magnitude of the corresponding cross-power spectral density are fed as input to neural networks for dialogue detection. Several types of artificial neural networks, including multilayer perceptrons, voted perceptrons, radial basis function networks, support vector machines, and particle swarm optimization-based multilayer perceptrons are tested. Experiments are carried out to validate the feasibility of the aforementioned approach by using ground-truth indicator functions determined by human observers on 6 different movies. A total of 41 dialogue instances and another 20 non-dialogue instances is employed. The average detection accuracy achieved is high, ranging between 84.78%±5.499% and 91.43%±4.239%.

Audio-Assisted Movie Dialogue Detection

“…The performance of dialogue detection of the proposed system is compared to the performance of a system that uses the ground truth indicator functions in both the training and the test phases [14]. In [14], two splits of the ground truth indicator functions between the training and the test set are examined, namely the 70%/30% training/test split and the 50%/50% training/test split.…”

“…In [14], two splits of the ground truth indicator functions between the training and the test set are examined, namely the 70%/30% training/test split and the 50%/50% training/test split. Concerning the RBF networks, for the 70%/30% split CCI is 0.872, while for the 50%/50% split CCI is 0.848.…”

“…The particular choice of the duration for the time window is justified in [14]. In short, after modeling the empirical distribution of the actor utterance duration, it is found that it is the Inverse Gaussian with expected value equal to 5 s. This means that actor changes are expected to occur, on average, every 5 s. We consider that four actor changes should occur within the time-window employed in our analysis on average.…”

“…Preliminary results on audio-assisted movie dialogue detection are described in [14] that resort to actor indicator functions. An actor indicator function defines if an actor speaks at a certain time instant.…”

Computationally Efficient and Robust BIC-Based Speaker Segmentation

“…The better the theoretical cdf approximates the empirical one, the closer the points (q i , F k ( t(i)−µ σ )) are to the diagonal [29]. To validate that IG distribution generally fits best the empirical speaker utterance duration distribution, another dataset has been employed, to be referred to as the movie dataset [30]. In this dataset, 25 audio recordings are included that have been extracted from six movies of different genres, namely: Analyze That, Cold Mountain, Jackie Brown, Lord of the Rings I, Platoon, and Secret Window.…”