LTC-SUM: Lightweight Client-driven Personalized Video Summarization Framework Using 2D CNN

Abstract

This paper proposes a novel lightweight thumbnail container-based summarization (LTC-SUM) framework for full featurelength videos. This framework generates a personalized keyshot summary for concurrent users by using the computational resource of the end-user device. State-of-the-art methods that acquire and process entire video data to generate video summaries are highly computationally intensive. In this regard, the proposed LTC-SUM method uses lightweight thumbnails to handle the complex process of detecting events. This significantly reduces computational complexity and improves communication and storage efficiency by resolving computational and privacy bottlenecks in resource-constrained end-user devices. These improvements were achieved by designing a lightweight 2D CNN model to extract features from thumbnails, which helped select and retrieve only a handful of specific segments.Extensive quantitative experiments on a set of full 18 feature-length videos (approximately 32.9 h in duration) showed that the proposed method is significantly computationally efficient than state-of-the-art methods on the same end-user device configurations. Joint qualitative assessments of the results of 56 participants showed that participants gave higher ratings to the summaries generated using the proposed method. To the best of our knowledge, this is the first attempt in designing a fully client-driven personalized keyshot video summarization framework using thumbnail containers for feature-length videos.

Personalized Video Summarization: A Comprehensive Survey of Methods and Datasets

“…• Dynamic summaries are also called dynamic video skims. Personalized video summarization is performed by selecting a subset of consecutive frames from all frames of the original video that includes the most relevant sub-shots that represent the original video according to the user's preferences as well as the summary criteria [12,14,[17][18][19][20]25,27,28,30,31,35,37,38,44,45,47,49,54,[59][60][61][62][63][64][65][66][67][68][69]. • A Hierarchical summary is a multilevel and scalable summary.…”

“…The unsupervised Multi-Graph Fusion (MGF) method was proposed by Ji et al [70] to find events that are automatically relevant to the query. The LTC-SUM method was proposed by Mujtaba et al [31] based on the design of a 2D CNN model to detect individual events through thumbnails. This model solves privacy and computation problems on end-user devices that are resource-constrained.…”

“…The system proposed by Kannan et al [66] can generate a video summarization without relying on any specific domain. The summaries presented in [12][13][14]17,20,22,25,27,30,31,33,34,45,47,51,54,55,[57][58][59][60][63][64][65][67][68][69][70][71]73] are not domain specific. The types of domains found in the literature are Personal Videos [52,53], Movies / serial clips [19,28,29,35,49], Sports Videos [18,35,[37][38][39][40][41][42][43][44],…”

“…The Sequential and Hierarchical Determinant Point Process (SH-DPP) is a probabilistic model developed by Sharghi et al [47] to be able to produce extractive summaries from streaming video or long-form video. The personalized video summaries presented in [12,13,17,18,24,31,37,38,45,49,59] are in real time. Figure 8 presents the distribution of the papers, reported in Table 2, between the time of the personalized summary.…”

“…Fei et al (2021) [30] proposed a triplet deep-ranking model for personalized video summarization. Mujtaba et al (2022) [31] proposed a framework for personalized video summarization using 2D CNN. öprü and Erzin (2022) [32] used Affective Visual Information for Human-Centric Video Summarization.…”