Mounting evidence suggests that environmental factors—in particular, those that we are exposed to during perinatal life—can dramatically shape the organism’s risk for later diseases, including neurobehavioral disorders. However, depending on the environmental insult, one sex may demonstrate greater vulnerability than the other sex. Herein, we focus on two well-defined extrinsic environmental factors that lead to sexually dimorphic behavioral differences in animal models and linkage in human epidemiological studies. These include maternal or psychosocial stress (such as social stress) and exposure to endocrine-disrupting compounds (such as one of the most prevalent, bisphenol A [BPA]). In general, the evidence suggests that early environmental exposures, such as BPA and stress, lead to more pronounced behavioral deficits in males than in females, whereas female neurobehavioral patterns are more vulnerable to later in life stress. These findings highlight the importance of considering sex differences and developmental timing when examining the effects of environmental factors on later neurobehavioral outcomes.