Rail surface defects seriously affect the safety of railway systems. At present, human inspection and rail vehicle inspection are the main approaches for the detection of rail surface defects. However, there are many shortcomings to these approaches, such as low efficiency, high cost, and so on. This paper presents a novel visual inspection approach based on unmanned aerial vehicle (UAV) images, and focuses on two key issues of UAV-based rail images: image enhancement and defects segmentation. With regards to the first aspect, a novel image enhancement algorithm named Local Weber-like Contrast (LWLC) is proposed to enhance rail images. The rail surface defects and backgrounds can be highlighted and homogenized under various sunlight intensity by LWLC, due to its illuminance independent, local nonlinear and other advantages. With regards to the second, a new threshold segmentation method named gray stretch maximum entropy (GSME) is presented in this paper. The proposed GSME method emphasizes gray stretch and de-noising on UAV-based rail images, and selects an optimal segmentation threshold for defects detection. Two visual comparison experiments were carried out to demonstrate the efficiency of the proposed methods. Finally, a quantitative comparison experiment shows the LWLC-GSME model achieves a recall of 93.75% for T-I defects and of 94.26% for T-II defects. Therefore, LWLC for image enhancement, in conjunction with GSME for defects segmentation, is efficient and feasible for the detection of rail surface defects based on UAV Images.