Surface defect detection based on computer vision remains a challenging task due to the uneven illumination, low contrast and miscellaneous patterns of defects. Current methods usually present undesirable detection accuracy and lack adaptability for the various scenes. In the paper, the novel uneven illumination surface defects inspection (UISDI) method is proposed to address these issues. First, the multi-scale saliency detection (MSSD) method is proposed to construct a coarse defect map and obtain the corresponding background regions. Second, a novel background similarity prior-based intrinsic image decomposition model (BSIID) is applied to divide the defect image into a non-defective shading layer and a defective reflectance layer. An accelerated optimization solution is proposed to solve the minimization problem of the intrinsic image decomposition model. Last, the enhanced defect image is obtained by filtering the reflectance image and is then utilized to accurately segment the defect region from the coarse defect map. The experiments conducted using four real-world defect datasets demonstrate that the proposed method outperforms state-of-the-art methods.