Multi-layer lightweight composite structures are widely used in the field of aviation and aerospace during the processes of manufacturing and use, and, as such, they inevitably produce defects, damage, and other quality problems, creating the need for timely non-destructive testing procedures and the convenient repair or replacement of quality problems related to the material. When using terahertz non-destructive testing technology to detect defects in multi-layer lightweight composite materials, due to the complexity of their structure and defect types, there are many signal characteristics of terahertz waves propagating in the structures, and there is no obvious rule behind them, resulting in a large gap between the recognition results and the actual ones. In this study, we introduced a U-Net-BiLSTM network that combines the strengths of the U-Net and BiLSTM networks. The U-Net network extracts the spatial features of THz signals, while the BiLSTM network captures their temporal features. By optimizing the network structure and various parameters, we obtained a model tailored to THz spectroscopy data. This model was subsequently employed for the identification and quantitative analysis of defects in multi-layer lightweight composite structures using THz non-destructive testing. The proposed U-Net-BiLSTM network achieved an accuracy of 99.45% in typical defect identification, with a comprehensive F1 score of 99.43%, outperforming the CNN, ResNet, U-Net, and BiLSTM networks. By leveraging defect classification and thickness recognition, this study successfully reconstructed three-dimensional THz defect images, thereby realizing quantitative defect detection.