Objective The aim of this study is to measure the motor development and tracking of physical fitness (PF) components of primary school children of Trier in Germany. Methods Two longitudinal cohorts, of 1768 children (915 f, 853 m) aged 5–11, were measured. In longitudinal cohort 1, a total of 116 female and 137 male participants aged 6.80 ± 0.42 years at baseline were measured four times from grade 1 to grade 4 (response: 40.4%). Participants of longitudinal cohort 2 (166 f, 149 m; 6.70 ± 0.36 years at baseline, response: 42.6%) were examined three times from grade 1 to grade 3 with the German Motor Test 6–18 (DMT 6–18). Results Physical fitness increased significantly over time in all test tasks except flexibility. Gender-specific differences were found in 20 m sprint, 6-minute run, balancing backwards, jumping sideways, and stand and reach. 74.4% of PF stability coefficients were moderate (r = 0.30 to 0.60). Stability of PF declined with increased time frames. Tracking was lower in girls than in boys. Flexibility showed the highest stability among PF variables (r > 0.50). BMI showed the overall highest stability coefficient with r > 0.7. Conclusions Gender-specific differences of PF were obvious but cannot always be secured statistically in primary school. Tracking was only moderate. Variability in the timing and speed of the adolescent growth spurt and sexual maturation influence stability of PF. Results from longitudinal cohort 2 largely confirm those from longitudinal cohort 1.