Background: We investigated a sample of cognitively healthy subjects with normal Alzheimer's disease (AD) cerebrospinal fluid (CSF) biomarker levels to identify the earliest variables related to longitudinal memory changes. Objective: Employing a new highly demanding learning and memory test (the Ancient Farming Equipment Test; AFE-T), we aimed to investigate whether a biomarker related to neurodegeneration (i.e., CSF tau) was associated with longitudinal memory decline.Methods: Thirty-two cognitively and biologically normal (CBN) subjects underwent MRI, neuropsychological assessment, and the AFE-T at baseline and 18 months later. To explore the relationship between cognitive performance and relevant factors, a linear model was set up. For a secondary analysis that further explore the effect of tau, the subjects were divided into CBN-Tau ↓ (tau < 228.64 pg/ml; n = 16) and CBN-Tau ↑ (tau > 228.64 pg/ml; n = 16). We also performed voxel-based morphometry (VBM) to identify regions of grey matter volume that would predict both baseline and longitudinal cognitive performance. Results: Our main finding was an association between CSF tau and longitudinal memory decline measured with AFE-T (B = -0.17, p < 0.05; r = -0.414; p < 0.01), and further analyses showed different evolvement between subgroups, with an accelerated decline in individuals with higher tau (F(1,31) = 8.37; p < 0.01). VBM results suggested that AFE-T performance is related to grey matter volume in a medial temporal, middle frontal, and posterior cerebellar network at baseline, and that there are strategic brain areas driving the longitudinal cognitive changes. Conclusions: The present findings provide evidence for structural and biological markers linked to cognitive aging by highlighting the role of tau, a marker of neurodegeneration, which can be related with the earliest memory changes in healthy subjects.