Males and females of almost all organisms exhibit sexual differences in body size, a phenomenon called sexual size dimorphism (SSD). How the sexes evolve to be different sizes, despite sharing the same genes that control growth and development, and hence a common genetic architecture, has remained elusive. Here, we show that the genetic architecture (heritabilities and genetic correlations) of the physiological mechanism that regulates size during the last stage of larval development of a moth, differs between the sexes, and thus probably facilitates, rather than hinders, the evolution of SSD. We further show that the endocrine system plays a critical role in generating SSD. Our results demonstrate that knowledge of the genetic architecture underlying the physiological process during development that ultimately produces SSD in adults can elucidate how males and females of organisms evolve to be of different sizes.