The rail surface images captured by line-scan cameras are susceptible to nonuniform illumination, stray light, smoothness variations of the rail surface, and so on, which degrade the detection accuracy of the rail surface spalling. To solve this problem, we propose an optical rail surface spalling detection algorithm based on visual saliency. First, the rail surface area is located to eliminate the interference of the surrounding area. Then, a two-dimensional difference of Gaussian (2D DoG) filter is used to reduce the noise. The filtered images are processed by means of a block local contrast measure (BLCM) estimator, which enhances the contrast of the spalling areas and produces the saliency map. Finally, a threshold is applied to locate the spalling areas. Experimental results demonstrate that the proposed algorithm achieves a detection accuracy of 93.5% and shows good robustness under nonuniform illumination and various rail surface smoothness conditions. Zhixin Hu (Non-member) is currently a Lecturer with the School of Mechanical and Vehicular Engineering, Nanchang Institute of Science and Technology, China. He is also pursuing the Ph.D. degree at the School of Mechanical and Electrical Engineering, Nanchang University. His research interests include measurement and control technology of optical mechatronics. Hongtao Zhu (Non-member) is now a Professor with the School of Mechanical and Electrical Engineering, Nanchang University. His research interests include measurement and control technology of optical mechatronics. Ming Hu (Non-member) received the M.Sc. degree