The detection of surface weak scratches is an intractable but vital task in optics-centered industries. However, the intrinsic characteristics of weak scratches, such as a narrow width, long span, and shallow depth, make it extremely difficult to effectively detect these scratches. In this paper, we tackle this issue from two perspectives. First, a multimodal microscopic imaging system is created by combining discrete multispectral illumination with linear polarization. Imaging experiments demonstrated that this system could highlight more scratch details, improve image clarity, and alleviate the image blur problem induced by wide spectrum scattered lights. Second, a scratch-oriented U-shaped deep encoder–decoder network equipped with optimized residual encoding modules, serial–parallel multiscale fusion modules, and triple-convolution decoding modules is proposed to segment the weak scratches from a raw image. The detection experiments demonstrate that our model can accurately segment the weak scratches on optical surfaces and achieve better detection performance using significantly fewer parameters compared to similar deep learning models. Meanwhile, experiments on the building crack dataset prove the excellent generalization capability.