Background: Little is known about the sex differences in the association between body mass index (BMI) and dementia in late life. Objective: Therefore, this retrospective cohort study aimed to analyze associations between BMI and dementia in older women and men separately in general practices in Germany. Methods: This study included patients followed in one of 832 general practices in Germany between 2006 and 2019 (index date: first visit date). Study variables included dementia (dependent variable), BMI (independent variable), age, sex, and comorbidities (control variables). Kaplan-Meier curves and adjusted Cox regression analyses were conducted to analyze associations between BMI and the 10-year incidence of dementia in women and men, separately. Results: There were 296,767 patients included in this study (mean [standard deviation] age 70.2 [5.9] years; 54.3% women). The proportion of underweight, normal weight, overweight, and obesity was 0.9%, 25.5%, 41.5%, and 32.1%, respectively. The 10-year incidence of dementia significantly decreased with increasing BMI, from 11.5% in women with underweight to 9.1% in those with obesity (log-rank p < 0.001). Respective figures in men were 12.0% and 8.2% (log-rank p < 0.001). In women, only overweight (versus normal weight) was significantly associated with dementia (HR = 0.93, 95% CI = 0.88–0.97). In contrast, in men, the only BMI category significantly associated with the incidence of dementia was underweight (HR = 1.58, 95% CI = 1.11–2.25). Conclusion: In this study conducted in Germany, overweight was negatively associated with dementia in women, whereas there was a positive underweight-dementia relationship in men. More data are needed to confirm or refute these findings in other settings.