The main objective of this study was to investigate the impact of prenatal and early postnatal stress on hippocampal volume in young adulthood. In sharp contrast to numerous results in animal models, our data from a neuroimaging follow-up (n = 131) of a community-based birth cohort from the Czech Republic (European Longitudinal Study of Pregnancy and Childhood) showed that in typically developing young adults, hippocampal volume was not associated with birth weight, stressful life events during the prenatal or early postnatal period, or dysregulated mood and wellbeing in the mother during the early postnatal period. Interestingly, mother’s anxiety/co-dependence during the first weeks after birth did show long-lasting effects on the hippocampal volume in young adult offspring irrespective of sex. Further analyses revealed that these effects were subfield-specific; present in CA1, CA2/3, CA4, GC-DG, subiculum, molecular layer, and HATA, hippocampal subfields identified by translational research as most stress- and glucocorticoid-sensitive, but not in the remaining subfields. Our findings provide evidence that the type of early stress is critical when studying its effects on the human brain.