Hierarchical Multimodal Attention for Deep Video Summarization

Abstract

The way people consume sports on TV has drastically evolved in the last years, particularly under the combined effects of the legalization of sport betting and the huge increase of sport analytics. Several companies are nowadays sending observers in the stadiums to collect live data of all the events happening on the field during the match. Those data contain meaningful information providing a very detailed description of all the actions occurring during the match to feed the coaches and staff, the fans, the viewers, and the gamblers. Exploiting all these data, sport broadcasters want to generate extra content such as match highlights, match summaries, players and teams analytics, etc., to appeal subscribers. This paper explores the problem of summarizing professional soccer matches as automatically as possible using both the aforementioned event-stream data collected from the field and the content broadcasted on TV. We have designed an architecture, introducing first (1) a Multiple Instance Learning method that takes into account the sequential dependency among events and then (2) a hierarchical multimodal attention layer that grasps the importance of each event in an action. We evaluate our approach on matches from two professional European soccer leagues, showing its capability to identify the best actions for automatic summarization by comparing with real summaries made by human operators.

Active Speaker Detection as a Multi-Objective Optimization with Uncertainty-Based Multimodal Fusion

“…Fig. 4 illustrates the concept of our fusion mechanism, inspired by the method presented in [17], that fuses the information from both modalities in a hierarchical fashion. First, we feed each video (u V t ) T t=1 and audio (u A t ) T t=1 embedded sequences to separated 128-dim BiGRU layers with hidden state h {V,A} t…”

“…In [16,17], the λ weights are learnt using perceptrons that fully-connect, at any given time, the hidden representation of each modality. This paper presents a novel approach to compute the weights of the modalities using: (1) an estimation of the uncertainty of both video and audio embedded representations, and (2) self-attention to measure the importance of video and audio modalities in their local temporal context.…”

“…Figures 5b and 5d compare the performance of our fusion method with the Naïve and Hierarchical [17] Fusion schemes in different scenarios. Our multi-objective model is first evaluated using a naïve concatenation fusion (NF) to combine video and audio modalities.…”

“…This result highlights the effectiveness of our end-to-end multi-objective learning. Hierarchical Fusion (HF) refers the fusion scheme presented in [17]. Note that our adaptation implies a slight modification of the initial method to match our "many-to-many" architecture.…”

“…Hore et al [16] proposed a model that selectively uses features from different modalities. This approach was later improved using hierarchical attention fusion in [17]. Despite the growing interest of these fusion mechanisms in other disciplines, to the best of our knowledge, no studies have been conducted on ASD problem.…”