There is substantial variation between healthy individuals in the number of retinal ganglion cells (RGC) in the eye, with commensurate variation in the number of axons in the optic tracts. Fixel-based analysis of diffusion MR produces estimates of fiber density (FD) and cross section (FC). Using these fixel measurements along with retinal imaging, we asked if individual differences in the estimated number of RGCs are correlated with individual differences in FD and FC measurements obtained from the optic tracts, and subsequent structures along the cortical visual pathway. We find that RGC endowment is correlated with optic tract FC, but not with FD. RGC cell number had a decreasing relationship with measurements from subsequent regions of the visual system (LGN volume, optic radiation FC/FD, and V1 surface area). However, we also found that the variations in each visual area were correlated with the variations in its immediately adjacent visual structure. We only observed these serial correlations when FC is used as the measure of interest for the optic tract and radiations, but no significant relationship was found when FD represented these white matter structures. From these results, we conclude that the variations in RGC endowment, LGN volume, and V1 surface area are better predicted by the overall cross section of the optic tract and optic radiations relative to the intra-axonal restricted signal component of these white matter pathways. Additionally, the presence of the significant correlations between adjacent, but not distant, anatomical structures suggest that there are multiple, local sources of anatomical variation along the visual pathway.