<b><i>Background:</i></b> The older population has been especially affected by the severe acute respiratory syndrome coronavirus 2 pandemic (COVID-19). <b><i>Objective:</i></b> The aim of the study was to explore the incidence, severity, mortality rate, clinical features, and risk factors of symptoms of COVID-19 in home-dwelling older people, and its association with type of residence, cognitive deterioration, and neurodegenerative diseases. <b><i>Methods:</i></b> Data about symptoms of COVID-19 were collected through a telephone survey in the cohort of 913 older volunteers of the Vallecas Project, aged 75–90 years, most of them (902) home-dwelling, in Madrid, Spain. The association of demographic and anthropometric measures, genetic polymorphisms, comorbidities, life habits, type of residence, and frailty surrogates were explored as potential risk factors for the incidence, severity, and mortality of COVID-19 in the older population. <b><i>Findings:</i></b> Sixty-two cases reported symptoms compatible with COVID-19; 6 of them had died, 4 in their home and 2 in the nursing home. Moderate/severe cases were significantly older and more frequently males. The <i>APOE</i> ε4 allele was associated with the presence of symptoms of COVID-19. Higher systolic blood pressure, more intense smoking habit, more alcohol intake, lower consumption of coffee and tea, and cognitive impairment were associated with disease severity. <b><i>Conclusions:</i></b> The estimated incidence of symptomatic COVID-19 in this older cohort of Madrid was 6.8%, with an overall mortality rate of 0.7% (18.2% in those living in a nursing home) and a fatality rate of 9.9%. Our exploratory study indicates that life habits, other clinical conditions and, the ε4 variant of the <i>APOE</i> gene are associated with the presence and clinical severity of coronavirus infection.