Common brain mechanisms are thought to play a significant role across a multitude of chronic pain syndromes. In addition, there is strong evidence for the existence of sex differences in the prevalence of chronic pain and in the neurobiology of pain. Thus, it is important to consider sex when developing general principals of pain neurobiology. The goal of the current review is to evaluate what is known about sex-specific brain alterations across multiple chronic pain populations. A total of 15 sex difference and 143 single-sex manuscripts were identified out of 412 chronic pain neuroimaging manuscripts. Results from sex difference studies indicate more prominent primary sensorimotor structural and functional alterations in female chronic pain patients compared to male chronic pain patients; differences in the nature and degree of insula alterations, with greater insula reactivity in male patients; differences in the degree of anterior cingulate structural alterations; and differences in emotional-arousal reactivity. Qualitative comparisons of male-specific and female-specific studies appear to be consistent with the results from sex difference studies. Given these differences, mixed-sex studies of chronic pain risk creating biased data or missing important information and single-sex studies have limited generalizability. The advent of large scale neuroimaging databases will likely aid in building a more comprehensive understanding of sex differences and commonalities in brain mechanisms underlying chronic pain.