Sex differences in neurological disease exist in incidence, severity, progression, and symptoms and may ultimately influence treatment. Cognitive disturbances are frequent in neuropsychiatric disease with men showing greater cognitive impairment in schizophrenia, but women showing more severe dementia and cognitive decline with Alzheimer's disease. Although there are no overall differences in intelligence between the sexes, men, and women demonstrate slight but consistent differences in a number of cognitive domains. These include a male advantage, on average, in some types of spatial abilities and a female advantage on some measures of verbal fluency and memory. Sex differences in traits or behaviors generally indicate the involvement of sex hormones, such as androgens and estrogens. We review the literature on whether adult levels of testosterone and estradiol influence spatial ability in both males and females from rodent models to humans. We also include information on estrogens and their ability to modulate verbal memory in men and women. Estrone and progestins are common components of hormone therapies, and we also review the existing literature concerning their effects on cognition. We also review the sex differences in the hippocampus and prefrontal cortex as they relate to cognitive performance in both rodents and humans. There has been greater recognition in the scientific literature that it is important to study both sexes and also to analyze study findings with sex as a variable. Only by examining these sex differences can we progress to finding treatments that will improve the cognitive health of both men and women. © 2016 American Physiological Society. Compr Physiol 6:1295-1337, 2016.