Spectral indices rarely show consistency in estimating crop traits across growth stages; thus, it is critical to simultaneously evaluate a group of spectral variables and select the most informative spectral indices for retrieving crop traits. The objective of this study was to explore the optimal spectral predictors for above-ground biomass (AGB) by applying Random Forest (RF) on three types of spectral predictors: the full spectrum, published spectral indices (Pub-SIs), and optimized spectral indices (Opt-SIs). Canopy hyperspectral reflectance of potato plants, treated with seven nitrogen (N) rates, was obtained during the tuber formation and tuber bulking from 2015 to 2016. Twelve Pub-SIs were selected, and their spectral bands were optimized using band optimization algorithms. Results showed that the Opt-SIs were the best input variables of RF models. Compared to the best empirical model based on Opt-SIs, the Opt-SIs based RF model improved the prediction of AGB, with R2 increased by 6%, 10%, and 16% at the tuber formation, tuber bulking, and for across the two growth stages, respectively. The Opt-SIs can significantly reduce the number of input variables. The optimized Blue nitrogen index (Opt-BNI) and Modified red-edge normalized difference vegetation index (Opt-mND705) combined with an RF model showed the best performance in estimating potato AGB at the tuber formation stage (R2 = 0.88). In the tuber bulking stage, only using optimized Nitrogen planar domain index (Opt-NPDI) as the input variable of the RF model produced satisfactory accuracy in training and testing datasets, with the R2, RMSE, and RE being 0.92, 208.6 kg/ha, and 10.3%, respectively. The Opt-BNI and Double-peak nitrogen index (Opt-NDDA) coupling with an RF model explained 86% of the variations in potato AGB, with the lowest RMSE (262.9 kg/ha) and RE (14.8%) across two growth stages. This study shows that combining the Opt-SIs and RF can greatly enhance the prediction accuracy for crop AGB while significantly reduces collinearity and redundancies of spectral data.