Multi-modal highlight generation for sports videos using an information-theoretic excitability measure

Abstract

The ability to detect and organize 'hot spots' representing areas of excitement within video streams is a challenging research problem when techniques rely exclusively on video content. A generic method for sports video highlight selection is presented in this study which leverages both video/image structure as well as audio/speech properties. Processing begins where the video is partitioned into small segments and several multi-modal features are extracted from each segment. Excitability is computed based on the likelihood of the segmental features residing in certain regions of their joint probability density function space which are considered both exciting and rare. The proposed measure is used to rank order the partitioned segments to compress the overall video sequence and produce a contiguous set of highlights. Experiments are performed on baseball videos based on signal processing advancements for excitement assessment in the commentators' speech, audio energy, slow motion replay, scene cut density, and motion activity as features. Detailed analysis on correlation between user excitability and various speech production parameters is conducted and an effective scheme is designed to estimate the excitement level of commentator's speech from the sports videos. Subjective evaluation of excitability and ranking of video segments demonstrate a higher correlation with the proposed measure compared to well-established techniques indicating the effectiveness of the overall approach.

Sports highlights generation bas ed on acoustic events detection: A rugby case study

“…The field of sports highlights detection or sports video summarization is extensively researched. Prior studies have used audio cues [1][2][3][4][5], visual cues [6], mixture of audio-visual cues [7][8], audio-textual cues [9], and a variety of features and classifiers to generate highlights [10]. Studies have also used textual cues from social media for generating sports highlights [11].…”

Engineering Analysis and Recognition of Nigerian English: An Insight into Low Resource Languages

“…The fundamental frequency of speech (F 0 ) is known to be affected by stress [19,20], emotions [19,21], and talking styles [22]. Different languages may exhibit unique F 0 characteristics [23] and the same may be observed also for individual dialects of a language [24].…”

Incorporating frequent pattern analysis into multimodal HMM event classification for baseball videos

“…Merler et al [30] proposed to use semantic model vectors, an intermediate level semantic representation, as a basis for modeling and detecting complex events in unconstrained real-world videos. Hasan et al [23] presented a generic video highlight generation scheme based on an information theoretic measure of user excitability. Liu et al [29] proposed to use action, scene and object concepts as semantic attributes for classification of video events.…”