Non-destructive stress measurement using ultrasonic techniques is based on calculation of the acoustoelastic coefficient obtained from the relationship between material stress and sound wave velocity. Longitudinal critically refracted (LCR) waves are the most suitable for ultrasonic stress measurement testing. This is because these waves exhibit a relatively large change in flight time in relation to a change in stress. In general, to calculate the acoustoelastic coefficient, tensile tests are performed with specimens extracted from the test objects to be inspected, and the difference in the speed of ultrasonic waves is measured in relation to the stress applied. However, train rails are usually subjected to compressive loads and no research has been conducted to confirm whether the results obtained using this method match the coefficients obtained from test objects in full cross-section. Therefore, results from compression tests on full cross-section specimens are needed as reference data to determine the reliability of the results obtained using extracted specimens. In this study, experiments were conducted to compare the result from applying compressive loads to full cross-section specimens to the experimental result obtained from specimens extracted from each rail section. For this purpose, ultrasonic probes capable of generating LCR waves were designed and manufactured to apply the technique to measure the applied stress. Based on the results obtained using this probe, the acoustoelastic coefficient values obtained from each specimen were compared. Then, the feasibility of calculating the internal stresses in railway rails using the acoustoelastic constants obtained from the extracted specimens was determined, and the reliability of the method was verified. It was concluded that there was no significant difference in the acoustoelastic coefficients calculated using the data from extracted specimens and those using full cross-section specimens.