Prenatal substance exposure (PSE) has been linked to adverse health outcomes, but its interactions with environmental and genetic factors remain unclear. Using data from the adolescent brain cognitive development cohort (n = 9,838; baseline age: 9.92 ± 0.62 years), we tested for the robust associations of PSE-caffeine/alcohol/tobacco/marijuana with children's health, cognition, and brain metrics after controlling for the environmental and genetic contexts. The environmental context involved birth, familial, and societal risk factors, while the genetic context included family histories and polygenic risk scores (PRSs) of mental disorders. In this sample, PSE-caffeine was observed in 59.8%, PSE-alcohol in 25.7%, PSE-tobacco in 13.2%, and PSE-marijuana in 5.6% of children. PSE-tobacco/marijuana was associated with higher environmental risks, PSE-alcohol was associated with lower familial risks, and all PSEs were associated with higher genetic risks. Controlling for these contexts reduced the number of significant health associations by 100, 91, 84, and 18% for PSE-tobacco/marijuana/caffeine/alcohol. Compared to the baseline, PSE-alcohol had the most health associations that were persistent over a 2-year period from preadolescence to adolescence, including associations with more sleep and mental health problems, improved cognitive functions, and larger brain volumes. These persistent associations with mental health problems and crystallized cognition were mediated by the surface areas of the frontal and the parietal cortices, respectively. Lower risk scores of the familial contexts attenuated associations between PSE-alcohol/marijuana and mental health problems. Higher PRS for substance use disorders enhanced late-onset associations of PSE-marijuana with externalizing problems. Results support the “health in context” concept, emphasizing modifiable factors mitigating adverse PSE effects.