The classification and quantitative description of damage of track structure for high-speed railway girder bridges (HRGBs) are still in the exploratory stage, which seriously interferes with the performance-based seismic damage assessment of HRGBs. In view of this, firstly, from the safety and stability of train operation, this paper defines the damage grade of track structure (intact, slight, moderate, severe and complete damage state) by refining the train operation index value. Secondly, based on the simple supported and continuous girder bridge structures of high-speed railway, the train-track-bridge system (TTBS) model is established, and the additional lateral irregularities of the rail caused by the lateral deform of the girder after the earthquake are induced. Finally, the influence of the lateral displacement amplitude of the rail and the train velocity on the train safety and stability is discussed. The results indicate that the rail lateral displacement by the reverse translation of the adjacent simply supported spans and the reverse rotation of the adjacent simply supported spans have the greatest impact on the train safety and stability, respectively. The lateral displacement of rail is proposed as an index to measure the damage of the track structure, and its critical value corresponding to each damage grade in the range of train velocity from 200km/h to 400km/h is determined, which can be used for the seismic vulnerability analysis of track structures for HRGBs.