The main purpose of the study was to evaluate the secular trends in physical fitness of Chinese rural children and adolescents aged 7–18 from 1985 to 2019. The speed, muscular strength, explosive power fitness, cardiorespiratory fitness, and flexibility were investigated by National Survey on Students’ Constitution and Health in 1985, 2000, 2010 and 2019. During the period 1985–2000, the physical fitness of Chinese rural children and adolescents improved except for flexibility, and almost all of them reached the maximum increase rate. During the period 2000–2010, in addition to the improvement of flexibility, other fitness showed downward trends, and the decline ranges were large. During the period 2010–2019, the speed of boys rebounded, flexibility, explosive power and muscular strength continued to decrease. Meanwhile, speed, flexibility and muscular strength in girls rebounded, and the explosive power continued to decline. From 2000 to 2019, the body mass index increase accelerated. This study shows that some components of physical fitness of Chinese rural children and adolescents have shown positive trends in recent years, especially for girls and adolescents aged 13–15 years. However, it may also imply inequality between sexes and ages, which provides a reference for the focus of the country's physical fitness and health monitoring and intervention measures.