Aim We investigated the association between levels of leisure-time physical activity and vascular stiffness in a longitudinal observational study from a representative Swedish population. Method A total of 2816 randomly selected individuals were examined at visit 1 (2002–2005, Men = 1400). After a mean follow-up of 9.7 ± 1.4 years, a representative sample of 1327 of the original participants were re-examined at visit 2. After excluding subjects with hypertension at baseline, 761 participants were included in the longitudinal analyses. Leisure-time physical (LTPA) activity was self-reported and dichotomized as high or low (level 3, 4 and level 1, 2, respectively). Large Arterial Elasticity Index (LAEI) and Small Arterial Elasticity Index (SAEI) were measured using the HDI/Pulse Wave™ CR2000. Multivariable general linear models were used to investigate the differences in changes SAEI and LAEI based on LTPA levels. Results At visit 1, and after adjustment for possible confounders, participants in the high LTPA group had better small artery elasticity (SAEI) (SAEI in low-level LTPA: 7.89 ± 0.11, SAEI in high-level LTPA: 8.32 ± 0.15, ΔSAEI: 0.42, CI: 0.07–0.78; p = 0.020). SAEI decreased between the two assessments (Visit 1: SAEI 8.01 ± 3.37 ml/mmHg; Δ SAEI: 1.4, CI 1.2–1.6, p < 0.001). Participants with a higher LTPA at visit 1 had significantly better SAEI at visit 2 (ΔSAEI: 0.44, CI 0.03–0.85, p = 0.037). No significant associations were observed between LAEI and LTPA after adjustments. Conclusions High LTPA predicted higher small arterial compliance at visit 2 suggesting that positive effects of LTPA on arterial elasticity persists over time.