Self-Attention Recurrent Summarization Network with Reinforcement Learning for Video Summarization Task

Summarizing Videos using Concentrated Attention and Considering the Uniqueness and Diversity of the Video Frames

“…In another recent work, Yaliniz et al [30] used Independently Recurrent Neural Networks (IndRNNs) [18] to model the temporal dependence of video frames, and learned summarization by using rewards associated with the representativeness, diversity and uniformity (i.e., the temporal coherence) of the video summary. Finally, Phaphuangwittayakul et al [21] presented a variation of the network architecture from [34], which estimates the frames' importance by combining the created representations for the video frames at the output of a bi-directional RNN and a self-attention mechanism.…”

“…where 𝜎 is the summary length regularization factor, a tunable hyper-parameter of our method (𝜎 was introduced in [19] and is used in several subsequent works, e.g., [3,17,21,34]). The computed training loss is then back-propagated to compute the gradients and update all the different parts of the architecture.…”

“…Finally, let us note that similarly to most of the existing works (e.g., [2,3,5,7,8,19,21,23,29,32,34]) we split each dataset into a training (80% of the videos) and a testing set (the remaining 20% of the videos) and we run experiments on five different randomlycreated splits for each dataset. In the following, we report the average performance over these runs (cf.…”

“…The reported values in Table 2 show that the proposed CA-SUM method performs consistently good on both of the utilized datasets, being the top-performing one on TVSum and the second best performing one on SumMe, according to the F-Score measure. In addition, our idea to work with a self-attention mechanism seems to be right, as five out of the six top-performing approaches ( [12,13,17,21]) contain some kind of attention mechanism. The use of more simple versions of Generator-Discriminator architectures ( [2,5,29]) cannot lead to competitive performance on the used datasets, given the current state of the art.…”

“…Nevertheless, these methods can suffer from the unstable training of Generative Adversarial Networks (GANs) [34], while their building blocks in most cases are different types of RNNs (mainly LSTMs), which show limited memorization capacity [17]. As an alternative, some works (e.g., [8,21,30,33,34]) design tailored reward functions that target specific desired characteristics of a good video summary (such as including representative and diverse content), and train a network architecture via reinforcement learning. However, the performance of most of these methods is poor, and they also carry the limitations of using RNNs to model long-range frames' dependencies [17].…”