The difference in body size between females and males, or sexual size dimorphism (SSD), is almost ubiquitous, and yet we have a remarkably poor understanding of the developmental-genetic mechanisms that generate it. Such an understanding is important if we are to distinguish between the many theoretical models of SSD evolution. One such model is the condition dependence hypothesis, which proposes that the body size of the larger sex is also more environmentally sensitive, a phenomenon called sex-specific plasticity (SSP). Because SSP generates differences in female and male body size, selection on plasticity may underlie the evolution of sexual size dimorphism. To test this hypothesis, however, we need to know the genetic architecture of both SSD and SSP, which is challenging because both are characteristics of populations not individuals. Here, we overcome this challenge by using isogenic lineages of Drosophila to measure both SSD and SSP for a genotype. We demonstrate extensive genetic variation for SSD among genotypes that is tightly correlated with variation in SSP, indicating that the same developmental-genetic mechanisms regulate both phenomena. These data support the condition dependence hypothesis and suggest that the observed SSD is a consequence of selection on the developmental-genetic mechanisms that regulate SSP.