Retinal optical coherence tomography has been identified as biomarker for disease progression in relapsing-remitting multiple sclerosis (RRMS), while the dynamics of retinal atrophy in progressive MS are less clear. We investigated retinal layer thickness changes in RRMS, primary and secondary progressive MS (PPMS, SPMS), and their prognostic value for disease activity. Here, we analyzed 2651 OCT measurements of 195 RRMS, 87 SPMS, 125 PPMS patients, and 98 controls from five German MS centers after quality control. Peripapillary and macular retinal nerve fiber layer (pRNFL, mRNFL) thickness predicted future relapses in all MS and RRMS patients while mRNFL and ganglion cell-inner plexiform layer (GCIPL) thickness predicted future MRI activity in RRMS (mRNFL, GCIPL) and PPMS (GCIPL). mRNFL thickness predicted future disability progression in PPMS. However, thickness change rates were subject to considerable amounts of measurement variability. In conclusion, retinal degeneration, most pronounced of pRNFL and GCIPL, occurs in all subtypes. Using the current state of technology, longitudinal assessments of retinal thickness may not be suitable on a single patient level.