We investigated which population groups were impacted most in terms of physical activity levels during the restrictions applied during the COVID-19 pandemic. We surveyed UK residents, sampled through users of a rewards-for-exercise app (Sweatcoin; n = 749) and an online panel (Prolific; n = 907). Of the app users, n = 487 further provided daily step-count data collected by the app, prior to, and during the periods of restrictions in the UK between March-June 2020. Regression models were used to investigate factors associated with self-reported change in physical activity and change in daily step-count during the periods of restrictions. Significant factors associated with self-reported change in physical activity included rural residents (positive, b = 0.87, p < 0.001), relative to urban dwellers, people classed as obese (negative, b = −0.51, p = 0.008, relative to healthy weight) and gym users (negative, b = −1.10, p < 0.001, relative to walkers). All groups had reduced step counts during restrictions, with Black, Asian and minority ethnic groups showing greater reductions compared to White British ethnicity (negative, b = −0.18, p = 0.008). Targeted interventions are required to ensure that physical and mental health impacts of sedentary behaviour are not exacerbated over the long-term by significant reductions in physical activity identified in these groups particularly those who are also more vulnerable to COVID-19.