Serotonin (5-HT) is a critical player in brain development and neuropsychiatric disorders. Fetal 5-HT levels can be influenced by several gestational factors, such as maternal genotype, diet, stress, medication, and immune activation. In this review, addressing both human and animal studies, we discuss how these gestational factors affect placental and fetal brain 5-HT levels, leading to changes in brain structure and function and behavior. We conclude that gestational factors are able to interact and thereby amplify or counteract each other’s impact on the fetal 5-HT-ergic system. We, therefore, argue that beyond the understanding of how single gestational factors affect 5-HT-ergic brain development and behavior in offspring, it is critical to elucidate the consequences of interacting factors. Moreover, we describe how each gestational factor is able to alter the 5-HT-ergic influence on the thalamocortical- and prefrontal-limbic circuitry and the hypothalamo-pituitary-adrenocortical-axis. These alterations have been associated with risks to develop attention deficit hyperactivity disorder, autism spectrum disorders, depression, and/or anxiety. Consequently, the manipulation of gestational factors may be used to combat pregnancy-related risks for neuropsychiatric disorders.