With the outbreak of the Corona Virus Disease 19 (COVID-19) in late 2019, governments increasingly imposed containment strategies, including social distancing as well as restricted population movement, potentially having negative impacts on mental and physical health. A growing number of studies have examined the impact of the pandemic on different facets of physical activity (PA); an overview combining these (mixed) results, however, is missing. Thus, the objective of this systematic review and meta-analysis was to investigate whether and to which extent PA changed from before to during the COVID-19 pandemic, taking age, gender, and measurement method into account. The literature search was conducted using PubMed, Web of Science, and Scopus. Results of the main characteristics were descriptively synthesized and analyzed in a meta-analysis quantifying effects of the pandemic on PA divided by age groups, with additional subgroup analyses of the characteristics age, gender, and measurement method being narratively synthesized. Overall, 57 studies with a total sample size of 119,094 participants (N between 10 and 60,560 subjects) from 14 countries worldwide with participants aged between four and 93 years were included. Thirty-two studies revealed a significant decline in PA, whereas only five studies found a significant increase in PA during the COVID-19 pandemic. Fourteen studies revealed mixed results. Except for children and adolescents, PA decreased in all age groups, independent of gender. Most self-reported and all device-based measurement methods showed a reduction in PA. However, effects were not found to be significant in all age groups. Nevertheless, the declining trend should be noted and governments should strive to enable PA within periods of pandemic restrictions, or promote alternatives such as digital training to avoid negative health consequences within the population.