Rail surface cracks are widespread damage that can lead to uneven surfaces of railheads and affect traveling safety. Non-destructive testing is needed to inspect rails regularly to ensure the normal operation of railroads. This paper proposes a laser ultrasonic testing method combining variational mode decomposition and diffractive Rayleigh wave time-of-flight to detect tiny cracks on the rail surface quantitatively. The finite element method was combined with experiments to simulate and experimentally investigate cracks of different sizes numerically. In the numerical simulation, the location of the crack was determined by B-scan. Afterward, the interaction between various types of ultrasound and cracks was comparatively analyzed, and the crack size was quantitatively characterized using useful information from the ultrasound signals. The results show that the time-of-flight method can detect arbitrary cracks with low error. Therefore, the experimentally acquired ultrasound signals used the time difference between the diffracted Rayleigh wave and other ultrasound waves to detect the crack information quantitatively. The variational mode decomposition method was used to separate the ultrasonic signals and extract the best surface wave modes to improve the signal-to-noise ratio. The results show that the combination of variational mode decomposition and time-of-flight method can effectively detect the size of cracks.