Changes in social and built environments most likely contribute to a decline in physical activity (PA) and physical fitness in children and adolescents. Organized sports may be an important component in ensuring adequate fitness, which is an important aspect in general health and well-being. The present study examines differences by club sports participation in cardiorespiratory endurance, muscular strength, power, speed, agility, flexibility and balance in 3293 (55.1% male) Austrian children and adolescents between 6 and 14 years of age. Anthropometric measurements (height and weight) were taken and participants completed the German motor test during regular class time. Even though there was no significant difference in body weight between club sports participants and non-club sports participants, club sports participation was associated with higher physical fitness, particularly regarding endurance, strength, power, and agility. Differences by club sports participation, however, declined during the elementary school years (6–10 years of age), while they became more pronounced during middle school years (10–14 years of age). Club sports participation, therefore, may be a viable option in the promotion of physical fitness, particularly during adolescence. At younger ages, other sources of PA, such as physical education and free play, however, should be considered to ensure sufficient fitness levels that contribute to a healthy and active lifestyle.