Memory and Perception-based Facial Image Reconstruction

Abstract

Visual memory for faces has been extensively researched, especially regarding the main factors that influence face memorability. However, what we remember exactly about a face, namely, the pictorial content of visual memory, remains largely unclear. The current work aims to elucidate this issue by reconstructing face images from both perceptual and memory-based behavioural data. Specifically, our work builds upon and further validates the hypothesis that visual memory and perception share a common representational basis underlying facial identity recognition. To this end, we derived facial features directly from perceptual data and then used such features for image reconstruction separately from perception and memory data. Successful levels of reconstruction were achieved in both cases for newly-learned faces as well as for familiar faces retrieved from long-term memory. Theoretically, this work provides insights into the content of memory-based representations while, practically, it opens the path to novel applications, such as computer-based 'sketch artists'.3Remembering the visual appearance of a known face is a crucial part of everyday life. To date, extensive research has established the impact of specific contextual and intrinsic facial properties on face memorability (e.g., distinctiveness, familiarity, inter-group similarity, race, emotional expression, and trustworthiness, to name a few) [1][2][3][4][5][6][7][8][9] . Yet, much less is currently known about the concrete pictorial information associated with retrieving a face from memory.Arguably, elucidating this issue can provide valuable insights into the nature of the representations subserving face memory and also, into their relationship with face perception. Accordingly, the current work seeks to elucidate the representational content of visual face memory through the novel use of image reconstruction. Previously, reconstruction approaches have been mainly directed at estimating the perceptual representations of an observer from patterns of neural activation 10-14 . Importantly though, reconstruction has not targeted longterm memory and its pictorial content as derived from behavioural data (but see recent work on neural-based image reconstruction from working memory 15 ). To handle this challenge, here, we appeal to a robust reconstruction approach 12 that capitalises on the structure of internal representations as reflected by empirical data irrespective of their modality (e.g., neural or behavioural). Further, this approach has a twofold goal of deriving facial features directly from empirical data and then using them in the process of image reconstruction.Theoretically, at the core of our work lies the concept of face space 16 , a multidimensional construct comprising a population of faces with the property that the distance between any pair of faces reflects their psychological similarity [17][18][19][20] . Critical for our purposes, perceptual face space and its memory-based counterpart may be closely related 21 allowing, in theory, the use of...

The Self in the Mind’s Eye: Revealing How We Truly See Ourselves Through Reverse Correlation

“…An important yet understudied constituent of the physical self is the mental representation of our body’s perceptual appearance (Pitron et al, 2018), including our size, shape, and facial characteristics (Carruthers, 2008). These are likely to be stored and retrieved in a pictorial, depictive format (Chang et al, 2017), essentially a mental picture of the self. How we picture ourselves in our mind’s eye has fundamental socioeconomical and clinical implications.…”

Modelling face memory reveals task-generalizable representations

“…There has been a recent surge of interest in modelling face representations from human memory [11][12][13]. These studies used 2D face images and applied dimensionality reduction (e.g.…”

“…pixel-wised RGB (or L*A*B) values. To understand the contribution of each 2D face space dimension to memory representations (including their neural coding), researchers modelled the relationship between projected weights of the original 2D face images on each dimension and participants' corresponding behavioral [13] (and brain [11,12]) responses.…”

Photogrammetry vs CT Scan: Evaluation of Accuracy of a Low‐Cost Three‐Dimensional Acquisition Method for Forensic Facial Approximation

“…CT-based 3D reproduction is often used for virtual facial approximation that allows the rebuilding of the facial soft tissues, considering specific tissue depth landmarks on the bony surface of the skull. In order to supply information for identification, forensic science research attempts to improve the accuracy of the technique (18)(19)(20)(21).…”