Bruise may cause spoilage, reduce commodity economic value, and give rise to food quality and safety concerns. Therefore, it is crucial to detect whether a loquat is bruised and when it is bruised to save storage and transportation costs. At present, the bruise of loquats is mainly discriminated by the operator’s naked eye, which is affected by personal habits, light intensity, and subjective psychological factors. The detection method is time-consuming, inaccurate, inefficient, and difficult to identify the bruise’s time of loquats. Due to the fact that the color features can be used to perform the conditions of the darkened and brownish regions in bruise’s loquats, the combined spectral information and the color features method is proposed to accurately detect the storage time of mild bruise’s loquats in this study. In order to reduce economic losses, different methods are used to deal with the loquats at the corresponding bruise’s time. Loquats with four types of bruise’s time, including 6, 12, 24, and 36 h, are studied. Models with four types of characteristics, including spectral information, RGB features combined with spectral information, HSI features combined with spectral information, and mixed color features combined with spectral information (mixed-spectral), are established based on linear discriminant analysis (LDA), support vector machine (SVM), and least-squares support vector machine (LS-SVM). The investigated 400 independent samples with four bruise’s time conditions are utilized to assess the classification ability of the proposed methods. The results indicate that the Mixed-RBF-LS-SVM model has the lowest errors, and the accuracies of storage time of mild bruise’s loquats at 6, 12, 24, and 36 h are 100%, 92%, 92%, and 100%, respectively. The overall accuracy of the LS-SVM model based on mixed-spectral is 96%, and it demonstrates that the combined spectral information and color features method can be used to accurately detect the bruise’s time of loquats. Finally, the LS-SVM model based on mixed-spectral is optimized by UVE, SPA, CARS, and GA, respectively; it is found that the UVE-LS-SVM model based on mixed-spectral is the best, and the overall accuracy is 92%. It also lays a foundation for future studies about detecting the bruise’s time of fruits with a high-precision, rapid, and nondestructive measurement.