Inherited retinal diseases can result from various genetic defects and are one of the leading causes for blindness in the working-age population. The present study aims to provide a comprehensive description of changes in retinal structure associated with phenotypic disease entities and underlying genetic mutations. Full macular spectral domain optical coherence tomography scans were obtained and manually segmented in 16 patients with retinitis pigmentosa, 7 patients with cone–rod dystrophy, and 7 patients with Stargardt disease, as well as 23 age- and sex-matched controls without retinal disease, to assess retinal layer thicknesses. As indicated by generalized least squares models, all IRDs were associated with retinal thinning (p < 0.001), especially of the outer nuclear layer (ONL, p < 0.001). Except for the retinal nerve fiber layer, such thinning was associated with a reduced visual acuity (p < 0.001). These advances in our understanding of ultrastructural retinal changes are important for the development of gene-, cell-, and optogenetic therapy. Longitudinal studies are warranted to describe the temporal component of those changes.