Effective soil spectral band selection and modeling methods can improve modeling accuracy. To establish a hyperspectral prediction model of soil organic matter (SOM) content, this study investigated a forested Eucalyptus plantation in Huangmian Forest Farm, Guangxi, China. The Ranger and Lasso algorithms were used to screen spectral bands. Subsequently, models were established using four algorithms: partial least squares regression, random forest (RF), a support vector machine, and an artificial neural network (ANN). The optimal model was then selected. The results showed that the modeling accuracy was higher when band selection was based on the Ranger algorithm than when it was based on the Lasso algorithm. ANN modeling had the best goodness of fit, and the model established by RF had the most stable modeling results. Based on the above results, a new method is proposed in this study for band selection in the early phase of soil hyperspectral modeling. The Ranger algorithm can be applied to screen the spectral bands, and ANN or RF can then be selected to construct the prediction model based on different datasets, which is applicable to establish the prediction model of SOM content in red soil plantations. This study provides a reference for the remote sensing of soil fertility in forests of different soil types and a theoretical basis for developing portable equipment for the hyperspectral measurement of SOM content in forest habitats.