Background: Alzheimer’s disease (AD) disproportionately affects females with steeper cognitive decline and more neuropathology compared to males, which is exacerbated in females carrying the APOE ɛ4 allele. The risk of developing AD is also higher in female APOE ɛ4 carriers in earlier age groups (aged 65–75), and the progression from cognitively normal to mild cognitive impairment (MCI) and to AD may be influenced by sex. Inflammation is observed in AD and is related to aging, stress, and neuroplasticity, and although studies are scarce, sex differences are noted in inflammation. Objective: The objective of this study was to investigate underlying physiological inflammatory mechanisms that may help explain why there are sex differences in AD and APOE ɛ4 carriers. Methods: We investigated, using the ADNI database, the effect of sex and APOE genotype (non-carriers or carriers of 1 and 2 APOE ɛ4 alleles) and sex and diagnosis (cognitively normal (CN), MCI, AD) on CSF (N = 279) and plasma (N = 527) markers of stress and inflammation. Results: We found CSF IL-16 and IL-8 levels were significantly lower in female non-carriers of APOE ɛ4 alleles compared to males, whereas levels were similar between the sexes among carriers of APOE ɛ4 alleles. Furthermore, females had on average higher levels of plasma CRP and ICAM1 but lower levels of CSF ICAM1, IL-8, IL-16, and IgA than males. Carrying APOE ɛ4 alleles and diagnosis (MCI and AD) decreased plasma CRP in both sexes. Conclusion: Sex differences in inflammatory biomarkers support that the underlying physiological changes during aging differ by sex and tissue origin.