In local disk galaxies such as our Milky Way, older stars generally inhabit a thicker disk than their younger counterparts. Two competing models have attempted to explain this result: one in which stars first form in thin disks that gradually thicken with time through dynamical heating, and one in which stars form in thick disks at early times and in progressively thinner disks at later times. We use a direct measure of the thicknesses of stellar disks at high redshift to discriminate between these scenarios. Using legacy Hubble Space Telescope imaging from the CANDELS and GOODS surveys, we measure the rest-optical scale heights of 491 edge-on disk galaxies spanning 0.4 ≤ z ≤ 2.5. We measure a median intrinsic scale height for the full sample of 0.74 ± 0.03 kpc, with little redshift evolution of both the population median and scatter. The median is consistent with the thick disk of the Milky Way today (0.6–1.1 kpc), but it is smaller than the median scale height of local disks (∼1.5 kpc) that are matched to our high-redshift sample by descendant mass. These findings indicate that, while (1) disks as thick as the Milky Way’s thick disk were in place at early times, (2) to explain the full disk galaxy population today, the stellar disks in galaxies need to on average physically thicken after formation.