Background: The COVID-19 pandemic negatively affected physical activity levels. This study investigated the factors associated with the change in physical activity level in Brazilians residing in the city of São Paulo. Methods: A self-administered questionnaire, addressing personal data, restriction level, education level, family income, daily working hours, and physical activity level, was answered by 2140 volunteers, of which 1179 were excluded because the answers were either incomplete or the respondents were not from São Paulo. The total number of participants selected was 961 (581 female and 380 male). Results: The physical activity level adopted prior to the pandemic period (p < 0.001) and family income (p = 0.001) correlated significantly with physical activity level reduction during the pandemic. The proportion of people who reduced their physical activity was greater among those who were very active than those who were active (adjusted prevalence ratio [aPR]: 0.65 [confidence interval (CI): 0.52–0.80]) or insufficiently active [aPR: 0.39 (0.18–0.82)]. The proportion of people who reduced their physical activity was greater among those who received a salary less than minimum wage (MW) than those who received a salary between three to six times minimum wage [(aPR: 0.50 (CI 0.35–0.70)] or more than 6 MW [(aPR: 0.56 (CI 0.40–0.79)]. Conclusions: A higher prevalence of Brazilians residing in the city of São Paulo reduced their physical activity who had a vigorous level of physical activity prior to the pandemic and who received less than a MW.