Little is known about the longitudinal trajectories and associations regarding physical activity (PA), physical fitness (PF), and cognitive skills in childhood. Accelerometer-based PA, sedentary behavior (SB), PF, and cognitive skills were measured in Estonian children (n = 147) in kindergarten (6.6 years) and again at school (7.6 years). Children were subgrouped into lower and upper quartiles by their moderate-to-vigorous PA (MVPA) at 6.6 years. Children in the upper quartile had lower SB, higher PA, and greater muscular strength. Attending school, MVPA in the lower quartile improved. In both subgroups, most strength values and cognitive skills improved, while balance deteriorated in first grade. In the upper quartile, a greater MPA at 6.6 years predicted lower perceptual skills at 7.6 years. A greater SB at 6.6 years predicted higher verbal skills, light and moderate PA and MVPA, and lower verbal skills at 7.6 years after taking into account confounding factors such as the child’s sex, age, awake wear time (AWT), maternal education, and/or child’s sports participation. A vigorous PA at 6.6 years predicted perceptual (in upper quartile) or verbal (in lower quartile) skills at 7.6 years after controlling for similar confounders. No correlation for PF at 6.6 years and cognitive skills at 7.6 years existed; after adjusting for the above-mentioned confounders relative to upper/lower-limb strength, the 4 × 10 m shuttle run results predicted higher perceptual or verbal skills; static balance and cardiorespiratory fitness predicted lower verbal skills. Cardiorespiratory fitness predicted higher perceptual skills after controlling for sex, age, and AWT. Overall, PA and strength were constantly better and SB lower in the upper quartile, yet the lower quartile demonstrated improved MVPA in first grade, and both subgroups increased most components of their strength and cognitive skills in first grade. Higher levels of VPA at kindergarten predicted either better perceptual or verbal skills in first grade after controlling for confounders; the opposite associations were found for other PA levels and cognitive skills in the higher quartile. PF components at kindergarten predicted either superior or inferior cognitive skills in first grade after adjusting for confounders.