Hyperspectral remote sensing can be applied to the rapid and nondestructive monitoring of heavy-metal pollution in crops. To realize the rapid and real-time detection of cadmium in the edible part (fruit) of Capsicum annuum, the leaf spectral reflectance of plants exposed to different levels of cadmium stress was measured using hyperspectral remote sensing during four growth stages. The spectral indices or bands sensitive to cadmium stress were determined by correlation analysis, and hyperspectral estimation models for predicting the cadmium content in the fruit of C. annuum during the mature growth stage were established. The models were cross validated by taking the sensitive spectral indices in the bud stage and the sensitive spectral bands in the flowering stage as the input variables. The results indicated that cadmium accumulated in the leaves and fruit of C. annuum and leaf cadmium content in the three early growth stages were correlated with the cadmium content of the pepper in the mature stage. Leaf spectral reflectance was sensitive to cadmium stress, and the first derivative of the original spectral reflectance was strongly correlated with leaf cadmium content during all growth stages. Among the established models, the multiple regression model based on the sensitive spectral bands in the flowering stage was optimal for predicting fruit cadmium content of the pepper. This model provides a promising method to ensure food safety during the early growth stage of the plant.