Most writing on sexual differentiation of the mammalian brain (including our own) considers just two organs: the gonads and the brain. This perspective, which leaves out all other body parts, misleads us in several ways. First, there is accumulating evidence that all organs are sexually differentiated, and that sex differences in peripheral organs affect the brain. We demonstrate this by reviewing examples involving sex differences in muscles, adipose tissue, the liver, immune system, gut, kidneys, bladder, and placenta that affect the nervous system and behavior. The second consequence of ignoring other organs when considering neural sex differences is that we are likely to miss the fact that some brain sex differences develop to compensate for differences in the internal environment (i.e., because male and female brains operate in different bodies, sex differences are required to make output/function more similar in the two sexes). We also consider evidence that sex differences in sensory systems cause male and female brains to perceive different information about the world; the two sexes are also perceived by the world differently and therefore exposed to differences in experience via treatment by others. Although the topic of sex differences in the brain is often seen as much more emotionally charged than studies of sex differences in other organs, the dichotomy is largely false. By putting the brain firmly back in the body, sex differences in the brain are predictable and can be more completely understood.