During the running of a high-speed train, the wheel may bounce on the rail due to the track irregularity. The wheel bounce could generate a vertical impact, leading to the initiation and expansion of delamination between layers of the track structure. In this paper, the evolution of the interfacial damage and delamination subjected to the vertical impact is simulated using finite element analysis (FEA). In the FEA, a bilinear cohesive zone model (CZM) is adopted to simulate the interface between the track slab and the CA mortar layer. For different levels of impact energy, the contact force, vertical deformation, absorbed energy, area of interfacial damage, and area of delamination are calculated and compared. The effects of the tangential and normal stiffness of the interface on the distribution of interfacial damage and delamination are investigated. The results show that the contact force, vertical deformation, absorbed energy, area of interfacial damage, and area of delamination increase with the increase of the impact energy. The area of interfacial damage in the compression stage is closely related to the tangential stiffness, whereas the area of delamination depends on the normal stiffness. The normal stiffness that gives the largest area of delamination is recommended to be taken as the lower bound of the normal stiffness for both controlling the delamination and preventing an exceedance of the track irregularity limit.