The accurate detection and quantification of defects is vital for the effectiveness of the eddy current nondestructive testing (ECNDT) of carbon fiber-reinforced plastic (CFRP) materials. This study investigates the identification and measurement of three common CFRP defects—cracks, delamination, and low-velocity impact damage—by employing the You Only Look Once (YOLO) model and an improved Eddy Current YOLO (EDC-YOLO) model. YOLO’s limitations in detecting multi-scale features are addressed through the integration of Transformer-based self-attention mechanisms and deformable convolutional sub-modules, with additional global feature extraction via CBAM. By leveraging the Wise-IoU loss function, the model performance is further enhanced, leading to a 4.4% increase in the mAP50 for defect detection. EDC-YOLO proves to be effective for defect identification and quantification in industrial inspections, providing detailed insights, such as the correlation between the impact damage size and energy levels.