Player performance in an intense sport such as basketball is known to be related to attributes such as speed, agility, and power. This study presents a comparative analysis of associations between anthropometric assessment and physical performance in different age groups of elite youth basketball players, while simultaneously identifying the predictors for speed and agility in these players. U14 (n = 44), U15 (n = 45), and U16 (n = 51) players were tested for anthropometry, lower-body power, speed, and agility. U16 players were found to be taller, heavier, more muscular than U14 and U15 players. In addition, the U16 group showed better performance in all performance tests. Age had a significant positive correlation with countermovement (CMJ) and drop jump (DJ) performance in U14 players, and a significant negative correlation with 15m and 20m sprint times in the U15 group. CMJ and DJ emerged as the most significant predictors for sprint and agility variables, respectively. Body fat percentage was found to be a significant predictor for the speed and agility tests in all age groups, but a negative lower-body power predictor. Therefore, besides all sport-specific and fitness tests, it is essential to place emphasis on the percentage of body fat when designing players’ individualized training programs, and during team selection.