Objectives Recent research has linked psychological (personality) factors and specific genetic risk polymorphisms to performance on neurocognitive phenotypes. We examined whether episodic or semantic memory performance is associated with (a) three personality traits (i.e., neuroticism, extraversion, openness to experience), (b) two neurodegenerative-related polymorphisms (i.e., Apolipoprotein E (APOE; rs7412; rs429358), Clusterin (CLU; rs11136000)), and (c) cross-domain risk interactions (magnification effects). Methods Linear growth models were examined to test independent associations between personality traits and declarative memory performance, and potential interaction effects with APOE and CLU genetic risk. Normal older adults (n = 282) with personality and genetic data from the Victoria Longitudinal Study were included at baseline and for up to 14 years of follow-up. Results First, we observed that higher openness to experience levels were associated with better episodic and semantic memory. Second, three significant gene × personality interactions were associated with poorer memory performance at baseline. These synergistic effects are: (a) APOE allelic risk (ε4+) carriers with lower openness to experience levels, (b) CLU (no risk: T/T) homozygotes with higher extraversion levels, and (c) CLU (no risk: T/T) homozygotes with lower neuroticism levels. Conclusions Specific neurodegenerative-related genetic polymorphisms (i.e., APOE and CLU) moderate and magnify the risk contributed by selected personality trait levels (i.e., openness to experience, extraversion) on declarative memory performance in non-demented aging. Future research could target interactions of other personality traits and genetic polymorphisms in different clinical populations for predicting other neurocognitive deficits or transitions to cognitive impairment and dementia.