It is now acknowledged that Alzheimer's Disease (AD) processes are present decades before the onset of clinical symptoms, but whether lifestyle activities can protect against these early AD processes in mid-life remains poorly understood. Furthermore, the impact of sex as a biological variable on associations between dementia risk, protective lifestyle activities and cognition is unknown. In this study, we aimed to replicate findings from our two recent studies [Deng et al. (2022) and Heneghan et al. (2022)] on the contribution of mid-life modifiable activities to cognition in individuals with dementia risk, in a larger independent cohort of the PREVENT–Dementia research program (N = 461 vs N = 208 used previously). Second, we investigated associations between biological sex, dementia risk, protective lifestyle activities and cognitive performance. Participants (40–59 years; N = 461) completed cognitive and clinical assessments cross-sectionally. Mid-life activities were measured with the Lifetime of Experiences Questionnaire. Known risk factors for sporadic late-onset AD (Apolipoprotein E ϵ4 allele status, family history of dementia, and the Cardiovascular Risk Factors Aging and Dementia score [CAIDE]) were investigated. Replicating our key previous findings (Deng et al., 2022 and Heneghan et al., 2022), we found that episodic and relational memory was (a) significantly negatively associated with the CAIDE risk score, (b) positively associated with stimulating lifestyle activities, and (c) that females performed significantly better than males in episodic and relational memory. The key novel finding of this study was that inherited dementia risk (i.e., APOE ϵ4 genotype) modulated the association between sex, lifestyle and cognition. Only for APOE ϵ4+ females, not APOE ϵ4-, higher occupational attainment was associated with better episodic and relational memory. Conversely, only for APOE ϵ4+ males, not APOE ϵ4-, higher occupational attainment was associated with worse episodic and relational memory. These findings suggest that modifiable lifestyle activities offset cognitive decrements due to inherited AD risk in mid-life and support the targeting of modifiable lifestyle activities for the prevention of Alzheimer's disease. Furthermore, these findings suggest an urgent need for targeted research on female-specific risk factors, to inform personalised strategies for AD prevention and the promotion of female brain health.