Early detection of defects inside the rail is of great significance to ensure the safety of rail transit. This work investigates the ability of ultrasonic guided waves (UGWs) to detect internal defects of the rail head. First, the model of UGWs propagation in rail, which has an irregular cross-section, is constructed based on semi-analytical finite element (SAFE). Fundamental characteristics such as wavenumber, phase or group velocity, and wave structure inside the rail are then calculated. Following modal and vibration energy distribution analysis, a guided wave mode sensitive to transverse fissure (TF) defects is selected and its excitation method is proposed. The effectiveness of the excitation method is confirmed by simulations performed in the ABAQUS software. According to the simulation data, the dispersion curve calculated by using the two-dimensional Fourier Fast Transform (2D-FFT) coincides well with that of SAFE. After that, the sensitivity of the selected mode to rail internal defect have been validated and its ability to locate defects has also been demonstrated. Finally, the effect of excitation frequency, defect size, defect vertical and horizontal depth on the reflection waveforms is investigated.