The present study strived to gain a more profound understanding of the distinctions in development between swimmers who are considered to be on track to the senior elite level compared to those who are not. Longitudinal data of 29 talented sprint and middle-distance swimmers (12 males; 17 females) on season best performances (season best times) and underlying performance characteristics (anthropometrics, starts, turns, maximal swimming velocity, stroke index [SI, an indirect measure of swimming efficiency] and lower body power) were collected over four swimming seasons (median of n = 3 seasons per swimmer). Based on their season best performance at early senior age (males aged 18–19; females aged 17–18), some swimmers were considered to be on track to reach the elite level (referred to as high-performing seniors; 6 males and 10 females), whereas others were not (referred to as lower-performing seniors; 6 males and 7 females). Retrospectively studying these swimmers (males and females separately), we found that all high-performing seniors were already on track to the elite level at late junior age (males aged 17; females aged 16), evidenced with faster season best performances throughout their transition compared to their lower-performing peers (p < 0.05). Independent sample t-tests revealed that high-performing seniors significantly outscored their lower-performing peers on maximal swimming velocity (males and females), starts and turns (males), SI (females) and lower body power (females) at late junior age (p < 0.05). Additionally, multilevel models showed faster rates of development for high-performing seniors on turns and maximal swimming velocity (males), and SI (females) compared to lower-performing peers during the junior-to-senior transition (p < 0.05). Particularly, the higher initial levels of swim performance and underlying characteristics at late junior age as well as the ability to keep progressing on season best performances (males and females), turns and maximal swimming velocity (males), and SI (females) during the junior-to-senior transition, may be crucial factors in the attainment of swimming expertise.