The COVID-19 lockdown has been associated with physical inactivity. We prospectively evaluated changes in moderate-to-vigorous physical activity (MVPA) and sedentary time (ST) among 1565 cardiovascular disease (CVD) patients using validated questionnaires at 5 weeks after lockdown initiation (i.e., baseline, April 2020) and at every 4 subsequent weeks, until July 2020. Multivariate mixed model analyses were performed to identify whether age, sex, CVD-subtype, lockdown adherence and mental health factors impacted changes in physical (in)activity. Patients were 67 (interquartile range: 60–73) years and primarily diagnosed with coronary artery disease. Time spent in MVPA was 143 min/day (95% confidence interval (CI) 137; 148) at baseline. Female sex, heart-failure, fear of COVID-19 infection and limited possibilities for physical activity were independently associated with lower levels of MVPA across time. After adjusting for confounders, overall MVPA did not change. ST was 567 (95% CI 555; 578) min/day at baseline. Lack of social contact, limited possibilities for physical activity and younger age were independently associated with higher levels of ST. After adjusting for confounders, ST progressively increased following 8 (Δ+19.7 (95% CI 0.4; 39.0)) and 12 weeks (Δ+25.2 (95% CI 5.4; 47.1) min/day) of lockdown. Despite a phased relaxation of the lockdown, CVD patients progressively increased ST and reported no change in MVPA. This highlights the need to target physical inactivity during and beyond the COVID-19 pandemic.