Understanding Image Memorability

Rust, Nicole C.; Mehrpour, Vahid

doi:10.1016/j.tics.2020.04.001

Cited by 76 publications

(58 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This discrepancy between machine and human intelligence was highlighted in a recent review (Rust & Mehrpour, 2020 ). More specifically, previous research has shown that neuronal activity for memorable versus non-memorable images is pooled together in the medial temporal lobe (Bainbridge et al, 2017 ) and in monkey inferotemporal cortex (Jaegle et al, 2019 ).…”

Section: Discussionmentioning

confidence: 99%

“…More specifically, previous research has shown that neuronal activity for memorable versus non-memorable images is pooled together in the medial temporal lobe (Bainbridge et al, 2017 ) and in monkey inferotemporal cortex (Jaegle et al, 2019 ). This finding suggests that memorable scene information might be very close in neuronal representational space, such that object identity is coded by neural spike pattern coding, and memorability is coded by spike magnitude coding (Rust & Mehrpour, 2020 ). In contrast, Lukavskẏ and Děchtěrenko ( 2017 ) showed that memorable scenes are more distant in a multidimensional space representing CNN-based image features.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

The effect of intrinsic image memorability on recollection and familiarity

Broers

Busch

2020

Mem Cogn

View full text Add to dashboard Cite

Many photographs of real-life scenes are very consistently remembered or forgotten by most people, making these images intrinsically memorable or forgettable. Although machine vision algorithms can predict a given image’s memorability very well, nothing is known about the subjective quality of these memories: are memorable images recognized based on strong feelings of familiarity or on recollection of episodic details? We tested people’s recognition memory for memorable and forgettable scenes selected from image memorability databases, which contain memorability scores for each image, based on large-scale recognition memory experiments. Specifically, we tested the effect of intrinsic memorability on recollection and familiarity using cognitive computational models based on receiver operating characteristics (ROCs; Experiment 1 and 1) and on remember/know (R/K) judgments (Experiment 1). The ROC data of Experiment 1 indicated that image memorability boosted memory strength, but did not find a specific effect on recollection or familiarity. By contrast, ROC data from Experiment 1, which was designed to facilitate encoding and, in turn, recollection, found evidence for a specific effect of image memorability on recollection. Moreover, R/K judgments showed that, on average, memorability boosts recollection rather than familiarity. However, we also found a large degree of variability in these judgments across individual images: some images actually achieved high recognition rates by exclusively boosting familiarity rather than recollection. Together, these results show that current machine vision algorithms that can predict an image’s intrinsic memorability in terms of hit rates fall short of describing the subjective quality of human memories.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

The effect of intrinsic image memorability on recollection and familiarity

Broers

Busch

2020

Mem Cogn

View full text Add to dashboard Cite

show abstract

“…Therefore, we believe that this effect may be due to high efficient face processing mechanisms (Crouzet et al, 2010), which could have led to more similarity in specific face-heavy categories (such as human and dog). Furthermore, as animacy could not fully explain arousal and valence effects on a behavioral and functional level of DCNNs, we argue that these could be a consequence of naturally learned optimization effects in visual systems, which have also been reported for image memorability judgments that automatically develop in DCNNs trained on object recognition and even predict variation in neural spiking activity (Jaegle et al, 2019;Rust and Mehrpour, 2020). This interpretation, however, needs to be treated with caution, as arousal and valence scores were obtained by human ratings which again underlie a wide range of effects such as for example acquired knowledge, attention, and memorability.…”

Section: Discussionmentioning

confidence: 87%

Comparing Object Recognition in Humans and Deep Convolutional Neural Networks—An Eye Tracking Study

Kwitt

Denzler

et al. 2021

Front. Neurosci.

View full text Add to dashboard Cite

Deep convolutional neural networks (DCNNs) and the ventral visual pathway share vast architectural and functional similarities in visual challenges such as object recognition. Recent insights have demonstrated that both hierarchical cascades can be compared in terms of both exerted behavior and underlying activation. However, these approaches ignore key differences in spatial priorities of information processing. In this proof-of-concept study, we demonstrate a comparison of human observers (N = 45) and three feedforward DCNNs through eye tracking and saliency maps. The results reveal fundamentally different resolutions in both visualization methods that need to be considered for an insightful comparison. Moreover, we provide evidence that a DCNN with biologically plausible receptive field sizes called vNet reveals higher agreement with human viewing behavior as contrasted with a standard ResNet architecture. We find that image-specific factors such as category, animacy, arousal, and valence have a direct link to the agreement of spatial object recognition priorities in humans and DCNNs, while other measures such as difficulty and general image properties do not. With this approach, we try to open up new perspectives at the intersection of biological and computer vision research.

show abstract

“…Our novel VR approach provides an ideal starting point for investigations into the factors and mechanisms that support memory usage in naturalistic settings. For example, manipulating the stimuli and their arrangements will inform the role of intrinsic memorability [58][59][60] when using memory. Further, increasing the granularity of the eye-movement recordings and changing the task relevance of location and identity object features 7,[22][23][24]26 will show whether using some features is costlier than others.…”

Section: Utilization Of Wm Representationsmentioning

confidence: 99%

When Natural Behavior Engages Working Memory

2021

View full text Add to dashboard Cite

Summary Working memory (WM) enables temporary storage and manipulation of information, 1 supporting tasks that require bridging between perception and subsequent behavior. Its properties, such as its capacity, have been thoroughly investigated in highly controlled laboratory tasks. 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 Much less is known about the utilization and properties of WM in natural behavior, 9 , 10 , 11 when reliance on WM emerges as a natural consequence of interactions with the environment. We measured the trade-off between reliance on WM and gathering information externally during immersive behavior in an adapted object-copying task. 12 By manipulating the locomotive demands required for task completion, we could investigate whether and how WM utilization changed as gathering information from the environment became more effortful. Reliance on WM was lower than WM capacity measures in typical laboratory tasks. A clear trade-off also occurred. As sampling information from the environment required increasing locomotion and time investment, participants relied more on their WM representations. This reliance on WM increased in a shallow and linear fashion and was associated with longer encoding durations. Participants’ avoidance of WM usage showcases a fundamental dependence on external information during ecological behavior, even if the potentially storable information is well within the capacity of the cognitive system. These foundational findings highlight the importance of using immersive tasks to understand how cognitive processes unfold within natural behavior. Our novel VR approach effectively combines the ecological validity, experimental rigor, and sensitive measures required to investigate the interplay between memory and perception in immersive behavior. Video Abstract

show abstract

Understanding Image Memorability

Cited by 76 publications

References 57 publications

The effect of intrinsic image memorability on recollection and familiarity

The effect of intrinsic image memorability on recollection and familiarity

Comparing Object Recognition in Humans and Deep Convolutional Neural Networks—An Eye Tracking Study

When Natural Behavior Engages Working Memory

Contact Info

Product

Resources

About