Quantitative data show that physical activity (PA) reduced during the COVID-19 pandemic, with differential impacts across demographic groups. Qualitative research is limited; thus, this study aimed to understand barriers and facilitators to PA during the pandemic, focusing on groups more likely to have been affected by restrictions, and to map these onto the capability, opportunity, motivation model of behaviour (COM-B). One-to-one interviews were conducted with younger (aged 18–24) and older adults (aged 70+), those with long-term physical or mental health conditions, and parents of young children. Themes were identified using reflexive thematic analysis and were mapped onto COM-B domains. A total of 116 participants contributed (aged 18–93, 61% female, 71% White British). Key themes were the importance of the outdoor environment, impact of COVID-19 restrictions, fear of contracting COVID-19, and level of engagement with home exercise. Caring responsibilities and conflicting priorities were a barrier. PA as a method of socialising, establishing new routines, and the importance of PA for protecting mental health were motivators. Most themes mapped onto the physical opportunity (environmental factors) and reflective motivation (evaluations/plans) COM-B domains. Future interventions should target these domains during pandemics (e.g., adapting PA guidance depending on location and giving education on the health benefits of PA).