The surface crack on track slab is the main cause of the service reliability deterioration of the high-speed rail (HSR) slab track. To realize the rapid and accurate detection of these cracks, infrared thermography (IRT) can be regarded as an effective way in HSR application. However, the performance of IRT can be significantly affected by the testing environment. This study aims to analyze the feasibility of IRT in crack detection for HSR slab track by investigating the key factors affecting the detection accuracy. Firstly, the principle of IRT in detecting surface crack of track slab is given, and the key factors influencing the detection performance are discussed. Considering the difficulty in conducting field tests for slab tracks, this study establishes a finite element (FE) model to simulate the temperature field of track slab with surface cracks. This model is verified by using the field measurement results of track slab temperature. With the FE model, the effects of detection time, ambient temperature, and sensitivity of the thermal imager on the detection accuracy are revealed. The results show that, during the window time for track maintenance (0:00 am–3:00 am), the difference of temperature between crack areas and noncrack areas gradually increases with time, so it is recommended to conduct detection during 2:00–3:00 am; under the condition of no auxiliary heat source, the crack detection is suggested to be arranged in summer with a high incidence of crack; the minimum limit of thermal sensitivity of thermal imager is below 108mk@25°C. The findings in this research can offer guidance for the implementation of IRT-based crack detection for HSR slab crack.