Zirconium sheet has been widely used in various fields, e.g., chemistry and aerospace. The surface scratches on the zirconium sheets caused by complex processing environment have a negative impact on the performance, e.g., working life and fatigue fracture resistance. Therefore, it is necessary to detect the defect of zirconium sheets. However, it is difficult to detect such scratch images due to lots of scattered additive noise and complex interlaced structural texture. Hence, we propose a framework for adaptively detecting scratches on the surface images of zirconium sheets, including noise removing and texture suppressing. First, the noise removal algorithm, i.e., an optimized threshold function based on dual-tree complex wavelet transform, uses selected parameters to remove scattered and numerous noise. Second, the texture suppression algorithm, i.e., an optimized relative total variation enhancement model, employs selected parameters to suppress interlaced texture. Finally, by connecting disconnection based on two types of connection algorithms and replacing the Gaussian filter in the standard Canny edge detection algorithm with our proposed framework, we can more robustly detect the scratches. The experimental results show that the proposed framework is of higher accuracy.