Skin defects are one of the main problems that occur during postharvest grading and processing of loquat, leading to deterioration of loquat, reducing the economic value of the commodity, and causing food quality and safety problems. Improving the identification rate of skin defects in loquat can reduce the economic loss caused by transportation and storage. In this paper, hyperspectral imaging technology was used to collect the reflectance (R), absorbance (A), and Kubelka-Munk (KM) spectra of loquat with skin defects for classification of defects types. Principal component analysis (PCA) was used to obtain the characteristic wavelength images. Competitive adaptive reweighted sampling (CARS), successive projections algorithm (SPA), uninformative variables elimination (UVE), and Monte Carlo combined with uninformative variables elimination (MCUVE) were used to reduce the dimension of spectral data to obtain the characteristic wavelength. The spectral data and grayscale features were used to establish the spectral model (SPEC) and grayscale features combined with spectral model (MIX). Extreme learning machine (ELM), least squares support vector machine (LS-SVM), and k-nearest neighbors (KNN) algorithm were applied to establish a classification model for skin defects in loquat. Comparing the model classification results of the three spectral parameters combined image features, it was found that the A-CARS-MIX-ELM model had the highest accuracy, with a classification accuracy of 98.18%. The number of selected characteristic spectra was 37, accounting for 21.02% of the total spectral number of the whole band. In the online detection process of a large number of fruits, we usually need to improve the detection speed on the premise of high-precision detection. In this case, the R-SPA-MIX-ELM model can be selected, and the classification accuracy is 94.55%. The number of selected spectra is 10, accounting for 5.68% of the number of wavelengths in the whole band. Consequently, it also provides a theoretical reference for the rapid, nondestructive, and high-precision fruit online detection technology in the future.