Topic-Oriented Text Features Can Match Visual Deep Models of Video Memorability

Abstract

Not every visual media production is equally retained in memory. Recent studies have shown that the elements of an image, as well as their mutual semantic dependencies, provide a strong clue as to whether a video clip will be recalled on a second viewing or not. We believe that short textual descriptions encapsulate most of these relationships among the elements of a video, and thus they represent a rich yet concise source of information to tackle the problem of media memorability prediction. In this paper, we deepen the study of short captions as a means to convey in natural language the visual semantics of a video. We propose to use vector embeddings from a pretrained SBERT topic detection model with no adaptation as input features to a linear regression model, showing that, from such a representation, simpler algorithms can outperform deep visual models. Our results suggest that text descriptions expressed in natural language might be effective in embodying the visual semantics required to model video memorability.

Value Assessment of UGC Short Videos through Element Mining and Data Analysis

“…1. Game Zone: Among the mainstream video divisions, the game partition exhibits the highest indicator weighting results in the creator and platform dimensions, suggesting that game videos in this category can generate more tangible benefits and value for creators and platforms [49]. However, compared to other partitions, the weighted results of indicators in the user dimension are average, indicating a relatively lower value contribution of game short videos to users.…”

ViPE: Visualise Pretty-much Everything

“…For instance, Opal (Liu et al, 2022c) enables structured search for visual concepts, Generative Disco (Liu et al, 2023a) facilitates text-tovideo generation for music visualisation, and Reel-Framer (Wang et al, 2023) aids in transforming written news stories into engaging video narratives for journalists. Nonetheless, despite their success at generating creative imagery, they still struggle to visualise figurative language effectively (Kleinlein et al, 2022;Chakrabarty et al, 2023;Akula et al, 2023). Furthermore, research by Chakrabarty et al (2023);Akula et al (2023) reveals that DALL•E 2 outperforms Stable Diffusion in representing figurative language.…”

“…In advertising, they frequently serve as persuasive tools to evoke positive attitudes (Phillips and McQuarrie, 2004;McQuarrie and Mick, 1999;Jahameh and Zibin, 2023). While humans effortlessly interpret images with metaphorical content (Yosef et al, 2023), state-of-the-art text-to-image models such as DALL.E 2 (Ramesh et al, 2022) and Stable Diffusion (Rombach et al, 2022) still struggle to synthesise meaningful images for such abstract and figurative expressions (Kleinlein et al, 2022;Chakrabarty et al, 2023;Akula et al, 2023).…”

Video Memorability Prediction From Jointly-learnt Semantic and Visual Features

“…Lately, the emphasis has been put on understanding the connection between the global semantics of an image (its visual constituent elements) and memorability. It has been shown that there exists a close correlation between certain topics and average memorability scores [12]. Therefore, even if many factors contribute to the memorability of a given sample, it seems that the main topic of a video (its semantic unit), extracted from text-based sources like captions, may be used as a proxy material to estimate its semantics and tackle the task of predicting memorability.…”

“…Recent studies from psychology and neurosciences seem to disagree with the idea that memory is an entirely subjective appraisal, instead suggesting that there are indeed visual elements that are more likely to be stored in memory for later recall [8,15,25]. Memorability is an observer-independent aspect of the visual medium, greatly influenced by the semantics of the scenes it represents [3], which motivates the use of alternative sources to analyse it beyond the purely visual domain, for instance, employing text-based captions that describe a scene [12].…”