An optimal early life environment is crucial for ensuring ideal neurodevelopmental outcomes. Brain development consists of a finely tuned series of spatially and temporally constrained events, which may be affected by exposure to a sub-optimal intra-uterine environment. Evidence suggests brain development may be particularly vulnerable to factors such as maternal nutrition, infection and stress during pregnancy. In this review, we discuss how maternal factors such as these can affect brain development and outcome in offspring, and we also identify evidence which suggests that the outcome can, in many cases, be stratified by socio-economic status (SES), with individuals in lower brackets typically having a worse outcome. We consider the relevant epidemiological evidence and draw parallels to mechanisms suggested by preclinical work where appropriate. We also discuss possible transgenerational effects of these maternal factors and the potential mechanisms involved. We conclude that modifiable factors such as maternal nutrition, infection and stress are important contributors to atypical brain development and that SES also likely has a key role.