Detecting defects on aerospace surfaces is critical to ensure safety and maintain the integrity of aircraft structures. Traditional methods often need more precision and efficiency for effective defect detection. This paper proposes an innovative approach that leverages deep learning and infrared imaging techniques to detect defects with high precision. The core contribution of our work lies in accurately detecting the size and depth of defects. Our method involves segmenting the size of the defect and calculating its centre to determine its depth. We achieve a more comprehensive and precise assessment of defects by integrating deep learning with infrared imaging based on the U-net model for segmentation and the CNN model for classification. The proposed model was rigorously tested on both a simulation dataset and an experimental dataset, demonstrating its robustness and effectiveness in accurately identifying and assessing defects on aerospace surfaces. The results indicate significant improvements in detection accuracy and computational efficiency, showing advancements over state-of-the-art methods and paving the way for enhanced maintenance protocols in the aerospace industry.