Traditionally sex hormones have been associated with reproductive and developmental processes only. Since the 1950s we know that hormones can have organizational effects on the developing brain and initiate hormonal transition periods such as puberty. However, recent evidence shows that sex hormones additionally structure the brain during important hormonal transition periods across a woman’s life including short-term fluctuations during the menstrual cycle. However, a comprehensive review focusing on structural changes during all hormonal transition phases of women is still missing. Therefore, in this review structural changes across hormonal transition periods (i.e., puberty, menstrual cycle, oral contraceptive intake, pregnancy and menopause) were investigated in a structured way and correlations with sex hormones evaluated. Results show an overall reduction in grey matter and region-specific decreases in prefrontal, parietal and middle temporal areas during puberty. Across the menstrual cycle grey matter plasticity in the hippocampus, the amygdala as well as temporal and parietal regions were most consistently reported. Studies reporting on pre- and post-pregnancy measurements revealed volume reductions in midline structures as well as prefrontal and temporal cortices. During perimenopause, the decline in sex hormones was paralleled with a reduction in hippocampal and parietal cortex volume. Brain volume changes were significantly correlated with estradiol, testosterone and progesterone levels in some studies, but directionality remains inconclusive between studies. These results indicate that sex hormones play an important role in shaping women’s brain structure during different transition periods and are not restricted to specific developmental periods.