The hollowing deterioration of stone relics required effective non-destructive testing (NDT) methods for their timely restoration and maintenance. To this end, a new NDT method based on terahertz (THz) technology by using support vector machine (SVM)-based machine learning models was developed to assess and diagnose the hollowing deterioration of the Yungang Grottoes. According to experiment design, a series of hollowing deterioration samples with various thicknesses of hollowing deterioration were prepared and then measured by using THz time-domain spectroscopy (THz-TDS). Based on the THz-TDS results of 30 randomly selected samples, a SVM-based hollowing deterioration prediction model (SVM-HDPM) was established by analyzing the relationship between the hollowing samples and the THz spectral information. The reliability and accuracy of the model was further proved by verified and compared with using the THz spectral data of the remaining 10 samples. The experimental results with the linear kernel function greatly demonstrated that the SVM-HDPM can have superior prediction accuracy, implying that the model is feasible for the prediction the hollowing deterioration of the stone relics. Moreover, one data preprocess was introduced into SVM-HDPM to meet the needs of field-based test. The predicted results of five different hollowing deterioration with different flaked stone thickness revealed good performance with very low mean square error (MSE) value. Therefore, it is believed that the proposed method can be regarded as an effective NDT technique with practical applications in analyzing cultural relics and have promising future prospects in inspection stone relics-like ancient heritage for hidden flaws.