Low physical fitness (PF) has been associated with higher risk of suffering from different diseases. The importance of PF is evident already in early ages, as children’s PF appears to be a key factor of their future PF and physical activity level. Among the variables that may have an influence on children’s PF, the importance of parent’s socioeconomic status and active/inactive behaviors has been stressed in several previous studies. However, previous literature has mostly reported this association through cross-sectional studies. The purpose of this study was to examine the impact of parental education and self-reported parental physical activity (PA) on their children’s development of PF during the 4-year duration of primary education. Using German Motor Test 6-18, the major components of PF (sprint velocity, coordination, flexibility, strength endurance, power, and endurance) were measured on a total of 371 children (46.9% girls, 30.6% migration background, 19.6% overweight/obese at the fourth test time point, compliance 70.1%) from 20 primary schools in Tyrol, Austria. Results showed that children with at least one parent with upper secondary education or above obtained significantly higher PF scores at all time points compared to children with both parents with lower secondary education and below. However, PF in both groups developed over time in a comparable manner irrespective of parental education. From the age of 9 years old, children with regularly physically active parents showed a stronger development of PF over the time compared to their peers with parents reporting irregular/no PA. Our results suggest that low-educated parents’ children might be considered a special target group for interventions aiming at increasing PF. More research is needed in order to delve into the potential underdevelopment of PF in 9-year-old children whose parents have low PA levels.