In order to realize the automatic classification of internal defects for non-contact nondestructive testing of concrete, a concrete multi-type defect classification algorithm based on the mixed strategy slime mold algorithm support vector machine (MSSMA-SVM) was proposed. The concrete surface’s vibration signal was obtained using a laser Doppler vibrometer (LDV) for four classification targets for no defect, segregation, cavity, and foreign matter concrete classification targets. The wavelet packet transform (WPT) decomposes the detected signals to get information on different frequency bands. The energy ratio change rate, energy ratio, and wavelet packet singular entropy of each node after the WPT were used as the feature input of MSSMA-SVM. The experimental results show that the designed MSSMA-SVM classifier can accurately detect the type, which provides a practical algorithm for classifying concrete defects by laser vibration measurement.