We assessed whether objectively measured low- and high-intensity physical activity (LPA and HPA) and sedentary time (ST) were associated with white matter connectivity, both throughout the whole brain and in brain regions involved in motor function. In the large population-based Maastricht Study (n = 1715, age 59.6 ± 8.1 (mean ± standard deviation) years, and 48% women), the amounts of LPA, HPA, and ST were objectively measured during 7 days by an activPAL accelerometer. In addition, using 3T structural and diffusion MRI, we calculated whole brain node degree and node degree of the basal ganglia and primary motor cortex. Multivariable linear regression analysis was performed, and we report standardized regression coefficients (stβ) adjusted for age, sex, education level, wake time, diabetes status, BMI, office systolic blood pressure, antihypertensive medication, total-cholesterol-to-HDL-cholesterol ratio, lipid-modifying medication, alcohol use, smoking status, and history of cardiovascular disease. Lower HPA was associated with lower whole brain node degree after full adjustment (stβ [95%CI] = − 0.062 [− 0.101, − 0.013]; p = 0.014), whereas lower LPA (stβ [95%CI] = − 0.013 [− 0.061, 0.034]; p = 0.580) and higher ST (stβ [95%CI] = − 0.030 [− 0.081, 0.021]; p = 0.250) was not. In addition, lower HPA was associated with lower node degree of the basal ganglia after full adjustment (stβ [95%CI] = − 0.070 [− 0.121, − 0.018]; p = 0.009). Objectively measured lower HPA, but not lower LPA and higher ST, was associated with lower whole brain node degree and node degree in specific brain regions highly specialized in motor function. Further research is needed to establish whether more HPA may preserve structural brain connectivity.