VideoMem: Constructing, Analyzing, Predicting Short-Term and Long-Term Video Memorability

Cohendet, Romain; Demarty, Claire-Hélène; Duong, Ngoc Q. K.; Engilberge, Martin

doi:10.1109/iccv.2019.00262

Cited by 34 publications

(47 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From Table IV we can see that our audio gestalt based approach outperforms all other approaches except SemanticMemNet [32]-the model introduced alongside the Memento10k dataset. [10] 0.552 Cohendet et al (ResNet3D) [10] 0.574 Feature Extraction + Regression (as in [43]) 0.615 SemanticMemNet [32] 0.663 Audio Gestalt 0.618* With respect to our results in Table III, the general trend for predicting video recognition memorability seems to be that the more modalities used, the better the predictions. Even the addition of a poorly-performing individual audio model (0.2913) with a better-performing individual visual model (0.4808), produces an increase in performance (0.4992).…”

Section: Resultsmentioning

confidence: 62%

“…The number of objects depicted in an image does not appear to directly relate to its overall recognition memorability [9], and likewise with properties such as aesthetics and interestingness [1]. However, combinations of semantically based attributes, such as object/scene category, emotion or actions, are predictive of recognition memorability [10]. Additionally, scrambled images retain consistencies in recognition memorability, but only for short time periods (seconds).…”

Section: A Visual Memorabilitymentioning

confidence: 98%

See 1 more Smart Citation

The Influence of Audio on Video Memorability with an Audio Gestalt Regulated Video Memorability System

Sweeney

Healy

Smeaton

2021

2021 International Conference on Content-Based Multimedia Indexing (CBMI)

View full text Add to dashboard Cite

Memories are the tethering threads that tie us to the world, and memorability is the measure of their tensile strength. The threads of memory are spun from fibres of many modalities, obscuring the contribution of a single fibre to a thread's overall tensile strength. Unfurling these fibres is the key to understanding the nature of their interaction, and how we can ultimately create more meaningful media content. In this paper, we examine the influence of audio on video recognition memorability, finding evidence to suggest that it can facilitate overall video recognition memorability rich in high-level (gestalt) audio features. We introduce a novel multimodal deep learning-based late-fusion system that uses audio gestalt to estimate the influence of a given video's audio on its overall short-term recognition memorability, and selectively leverages audio features to make a prediction accordingly. We benchmark our audio gestalt based system on the Memento10k short-term video memorability dataset, achieving top-2 state-of-the-art results.

show abstract

Section: Resultsmentioning

confidence: 62%

Section: A Visual Memorabilitymentioning

confidence: 98%

The Influence of Audio on Video Memorability with an Audio Gestalt Regulated Video Memorability System

Sweeney

Healy

Smeaton

2021

2021 International Conference on Content-Based Multimedia Indexing (CBMI)

View full text Add to dashboard Cite

show abstract

“…Furthermore, every video must also have a memorability score attached indicating its degree of memorability. Amidst the corpora accessible by the research community that meet our criteria, two in particular stand out: VideoMem and Memento10K [19,20]. Although it is true that data are extracted from different sources and their annotation procedures are not exactly equal, in this study we shall consider that both sets of labels represent the same concept closely enough to model them following the very same principles.…”

Section: Datasetsmentioning

confidence: 99%

“…Videos are never presented in the same fixed order, but randomly. Because it has been observed that video memorability depends in a linear way on the number of videos between two occurrences [19,20], in order to homogenize memorability labels among videos a linear correction (firstly introduced in [16]) is applied to the raw hit scores, thus obtaining the final set of labels.…”

Section: Videomemmentioning

confidence: 99%

Topic-Oriented Text Features Can Match Visual Deep Models of Video Memorability

et al. 2021

View full text Add to dashboard Cite

Not every visual media production is equally retained in memory. Recent studies have shown that the elements of an image, as well as their mutual semantic dependencies, provide a strong clue as to whether a video clip will be recalled on a second viewing or not. We believe that short textual descriptions encapsulate most of these relationships among the elements of a video, and thus they represent a rich yet concise source of information to tackle the problem of media memorability prediction. In this paper, we deepen the study of short captions as a means to convey in natural language the visual semantics of a video. We propose to use vector embeddings from a pretrained SBERT topic detection model with no adaptation as input features to a linear regression model, showing that, from such a representation, simpler algorithms can outperform deep visual models. Our results suggest that text descriptions expressed in natural language might be effective in embodying the visual semantics required to model video memorability.

show abstract

“…The second point, closely related to the previous one, is the lack of a common definition for VM. Regarding modelling, previous attempts at predicting VM [3,12] have highlighted several features which contribute to the prediction of VM, such as semantic, saliency and colour features, but the work is far from complete and our capacity to propose effective computational models will help to meet the challenge of VM prediction. The goal of this task is to participate in the harmonisation and the advancement of this emerging multimedia field.…”

Section: Introductionmentioning

confidence: 99%

Increasing Image Memorability with Neural Style Transfer

Siarohin

Zen

Majtanovic

et al. 2019

ACM Trans. Multimedia Comput. Commun. Appl.

View full text Add to dashboard Cite

Recent works in computer vision and multimedia have shown that image memorability can be automatically inferred exploiting powerful deep learning models. This paper advances the state of the art in this area by addressing a novel and more challenging issue: "Given an arbitrary input image, can we make it more memorable?". To tackle this problem we introduce an approach based on an editing-by-applying-filters paradigm: given an input image, we propose to automatically retrieve a set of "style seeds", i.e. a set of style images which, applied to the input image through a neural style transfer algorithm, provide the highest increase in memorability. We show the effectiveness of the proposed approach with experiments on the publicly available LaMem dataset, performing both a quantitative evaluation and a user study. To demonstrate the flexibility of the proposed framework, we also analyze the impact of different implementation choices, such as using different state of the art neural style transfer methods. Finally, we show several qualitative results to provide additional insights on the link between image style and memorability.

show abstract

VideoMem: Constructing, Analyzing, Predicting Short-Term and Long-Term Video Memorability

Cited by 34 publications

References 25 publications

The Influence of Audio on Video Memorability with an Audio Gestalt Regulated Video Memorability System

The Influence of Audio on Video Memorability with an Audio Gestalt Regulated Video Memorability System

Topic-Oriented Text Features Can Match Visual Deep Models of Video Memorability

Increasing Image Memorability with Neural Style Transfer

Contact Info

Product

Resources

About