Prenatal stress is associated with child behavioral outcomes increasing susceptibility for psychiatric disorders in later life. Altered fetal brain development might partly mediate this association, as some studies suggest. With this study, we investigated the relation between prenatal stress, child’s brain structure and behavioral problems. The association between self-reported maternal pregnancy-related anxiety (PRAQ-R2 questionnaire, second and third trimester) and brain gray matter volume was probed in 27 4-year-old children (13 female). Voxel based morphometry was applied with an age-matched template in SPM for the whole-brain analyses, and amygdala volume was assessed with manual segmentation. Possible pre- and postnatal confounders, such as maternal depression and anxiety among others, were controlled for. Child behavioral problems were assessed with the Strength and Difficulties Questionnaire by maternal report. We found a significant interaction effect of pregnancy-related anxiety and child’s sex on child’s amygdala volume, i.e., higher pregnancy-related anxiety in the second trimester was related to significantly greater left relative amygdala volume in girls compared to boys. Further exploratory analyses yielded that both maternal pregnancy-related anxiety and child’s amygdala volume are related to child emotional and behavioral difficulties: While higher pregnancy-related anxiety was associated with more emotional symptoms, peer relationship problems and overall child difficulties, greater left amygdala volume was related to less of these child difficulties and might partly mediate sex-specific associations between pregnancy-related anxiety and child behavioral difficulties. Our data suggest that maternal prenatal distress leads to sexually dimorphic structural changes in the offspring’s limbic system and that these changes are also linked to behavioral difficulties. Our results provide further support for the notion that prenatal stress impacts child development.