Background Cognitive impairment (CI) is a frequent and debilitating symptom in MS. To better understand the neural bases of CI in MS, this magnetic resonance imaging (MRI) study aimed to identify and quantify related structural brain changes and to investigate their relation to each other. Methods We studied 51 patients with CI and 391 patients with cognitive preservation (CP). We analyzed three-dimensional T1-weighted and FLAIR scans at 3 Tesla. We determined mean cortical thickness as well as volumes of cortical grey matter (GM), deep GM including thalamus, cerebellar cortex, white matter, corpus callosum, and white matter lesions (WML). We also analyzed GM across the whole brain by voxel-wise and surface-based techniques. Results Mean disease duration was 5 years. Comparing MS patients with CI and CP, we found higher volumes of WML, lower volumes of deep and cortical GM structures, and lower volumes of the corpus callosum (all corrected p values < 0.05). Effect sizes were largest for WML and thalamic volume (standardized ß values 0.25 and − 0.25). By logistic regression analysis including both WML and thalamic volume, we found a significant effect only for WML volume. Inclusion of the interaction term of WML and thalamic volume increased the model fit and revealed a highly significant interaction of WML and thalamic volume. Moreover, voxel-wise and surface-based comparisons of MS patients with CI and CP showed regional atrophy of both deep and cortical GM independent of WML volume and overall disability, but effect sizes were lower. Conclusion Although several mechanisms contribute to CI already in the early stage of MS, WML seem to be the main driver with thalamic atrophy primarily intensifying this effect.