Background: The association between physical activity (PA) and arterial stiffness is particularly important in children with congenital heart disease (CHD) who are at risk for arterial stiffening. The aim of this study was to examine the association between objectively measured PA and arterial stiffness in children and adolescents with CHD. Methods: In 387 children and adolescents with various CHD (12.2 ± 3.3 years; 162 girls) moderate-to-vigorous PA (MVPA) was assessed with the “Garmin vivofit jr.” for 7 consecutive days. Arterial stiffness parameters including pulse wave velocity (PWV) and central systolic blood pressure (cSBP) were non-invasively assessed by oscillometric measurement via Mobil-O-Graph®. Results: MVPA was not associated with PWV (ß = −0.025, p = 0.446) and cSBP (ß = −0.020, p = 0.552) in children with CHD after adjusting for age, sex, BMI z-score, peripheral systolic blood pressure, heart rate and hypertensive agents. Children with CHD were remarkably active with 80% of the study population reaching the WHO recommendation of average 60 min of MVPA per day. Arterial stiffness did not differ between low-active and high-active CHD group after adjusting for age, sex, BMI z-score, peripheral systolic blood pressure, heart rate and hypertensive agents (PWV: F = 0.530, p = 0.467; cSBP: F = 0.843, p = 0.359). Conclusion: In this active cohort, no association between PA and arterial stiffness was found. Longer exposure to the respective risk factors of physical inactivity might be necessary to determine an impact of PA on the vascular system.