This paper proposes a new approach to classify and evaluate defects in concrete structures automatically. To overcome the limitations of defect detection methods that traditionally relied on expert visual observation, the reflection signal of electromagnetic pulses is extracted as time-series data and used to analyze the propagation characteristics of each defect. This study uses deep learning models to analyze these time-series data and classify defects. Since anomaly detection data has more normal data than anomaly data, data augmentation methods such as Time Warping, Noise Injection, Smoothing, Trend Shifting, etc., were applied to solve the problem of data imbalance and overfitting. Among them, Noise Injection showed the best performance. The generalization performance of the proposed method was evaluated through performance evaluation using LSTM, GRU, and TCN models, and LSTM models showed the highest performance. The study results show that the proposed method effectively classifies defect types in concrete structures and can solve the limitations of existing methods by automatic classification through deep learning models. In addition, it was confirmed that the model's performance could be improved by improving the amount and diversity of data by selecting and applying appropriate data augmentation methods. The contribution of the research is to present a new approach that automates the defect detection and classification task of concrete structures and provides high accuracy and efficiency.