Chaenomelis Fructus is a widely used traditional Chinese medicine with a long history in China. The total content of oleanolic acid (OA) and ursolic acid (UA) is taken as an important quality marker of Chaenomelis Fructus. In this study, quantitative models for the prediction total content of OA and UA in Chaenomelis Fructus were explored based on near-infrared spectroscopy (NIRS). The content of OA and UA in each sample was determined using high-performance liquid chromatography (HPLC), and the data was used as a reference. In the partial least squares (PLS) model, both leave one out cross validation (LOOCV) of the calibration set and external validation of the validation set were used to screen spectrum preprocessing methods, and finally the multiplicative scatter correction (MSC) was chosen as the optimal pretreatment method. The modeling spectrum bands and ranks were optimized using PLS regression, and the characteristic spectrum range was determined as 7,500–4,250 cm−1, with 14 optimal ranks. In the back propagation artificial neural network (BP-ANN) model, the scoring data of 14 ranks obtained from PLS regression analysis were taken as input variables, and the total content of OA and UA reference values were taken as output values. The number of hidden layer nodes of BP-ANN was screened by full-cross validation (Full-CV) of the calibration set and external validation of the validation set. The result shows that both PLS model and PLS-BP-ANN model have strong prediction ability. In order to evaluate and compare the performance and prediction ability of models, the total content of OA and UA in each sample of the test set were detected under the same HPLC conditions, the NIRS data of the test set were input, respectively, to the optimized PLS model and PLS-BP-ANN model. By comparing the root-mean-square error (RMSEP) and determination coefficient (R2) of the test set and ratio of performance to deviation (RPD), the PLS-BP-ANN model was found to have better performance with RMSEP of 0.59 mg·g−1, R2 of 95.10%, RPD of 4.53 and bias of 0.0387 mg·g−1. The results indicated that NIRS can be used for the rapid quality control of Chaenomelis Fructus.