Previous studies examining the association between kidney stone disease (KSD) and arterial stiffness have been limited. Both age and gender have been found to have an impact on KSD, but their influence on the relationship between KSD and increased arterial stiffness is unclear. This study included 6694 subjects from October 2006 to August 2009. The diagnosis of kidney stone was based on the results of ultrasonographic examination. Increased arterial stiffness was defined as right-sided brachial-ankle pulse wave velocity (baPWV) ≥ 14 m/s. Associations between KSD and increased arterial stiffness were analyzed using multiple logistic regression models. KSD was positively related to increased arterial stiffness in both male and female groups (males: odds ratio [OR], 1.306; 95% confidence interval [CI], 1.035–1.649; females: OR, 1.585; 95% CI, 1.038–2.419) after adjusting for confounding factors. Subgroup analysis by age group (<50 and ≥50 years) showed a significant positive relationship only in the groups ≥ 50 years for both genders (males: OR, 1.546; 95% CI, 1.111–2.151; females: OR, 1.783; 95% CI, 1.042–3.054), but not in the groups < 50 years. In conclusion, KSD is associated with a higher risk of increased arterial stiffness in individuals aged ≥ 50 years, but not in those aged < 50 years for both genders.