Physical fitness (PF) of children and adolescents is an important resource for their future health. Population-based studies, however, rarely report secular changes of PF, although monitoring of these is crucial to deriving information for adequate interventions. This review aims to report trends in PF of children and adolescents. A literature search was conducted in PubMed in July 2019. Cohort studies published in English allowing statements to be made on trends in PF by comparing youth between the ages of four and 18 years were included. The review identified 24 studies from 16 countries meeting the inclusion criteria, with an overall sample size of more than 860,000 children and adolescents. Through a standardized quality assessment tool, we classified two studies as strong, 21 as moderate, and only one as weak. We analyzed specific secular trends separately for the five different dimensions: endurance, strength, speed, flexibility, and coordination. The majority of studies report a decline of PF over time; however, a few studies report conflicting results. Performance in endurance, strength, and flexibility decreased over time, whereas there was no consistent trend reported for speed and coordination. Overall, there is no international standard on examining and reporting changes or secular trends in PF of children and adolescents, and comparability of studies is limited due to heterogeneous conditions of conducting and analyzing PF tests. Consequently, standardized and consistent international monitoring should be implemented.