This article proposes a progressive frequency domain-guided depth model with adaptive preprocessing to solve the problem of defect detection with weak features based on X-ray images. In distinct intuitive surface defect detection tasks, non-destructive testing of castings using X-rays presents more complex and weak defect features, leading to lower accuracy and insufficient robustness on the part of current casting defect detection methods. To address these challenges, the proposed method establishes four specialized mechanisms to improve model accuracy. First, an adaptive image contrast enhancement method is proposed to enhance the features of defects in casting images to promote subsequent feature extraction and prediction. Second, a subtle clue mining module based on frequency domain attention is proposed to fully extract the discriminative features of casting defects. Third, a feature refinement module based on progressive learning is proposed to achieve a balance between feature resolution and semantic information. Finally, a refined deep regression supervision mechanism is designed to improve defect detection accuracy under strict intersection-to-union ratio standards. We established extensive ablation studies using casting defect images in GDXray, conducted detailed comparative experiments with other methods, and performed experiments to analyze the robustness of the resulting models. Compared with other X-ray defect detection methods, our framework achieves an average +4.6 AP. Compared to the baseline, our proposed refined deep regression supervision mechanism results in an improvement of 5.3 AP.