Rail fastener is a crucial component equipment to ensure the safe operation of the train, and it is very paramount to detect the loose state of the fastener. In this paper, the vertical vibration acceleration signal of wheelset is taken as the research object, and the loose state of fastener is identified by separating and calculating the key IMF energy entropy. Firstly, based on the finite element theory and the principle of multibody dynamics, the rigid-flexible coupling simulation model of vehicle track is established. Then, the vertical vibration acceleration signals of the wheelset under the speed of 200 km/h are obtained by setting the different loosening degrees of the fastener. Finally, we use optimized HHT to process signals, and the orthogonal empirical mode decomposition method (OEMD) is proposed to optimize the orthogonality of the intrinsic mode function, to eliminate the IMF component having poor correlation with the original signal; the Hilbert time spectrum and information entropy theory are combined to calculate the energy entropy of the key IMF, and the HHT energy entropy evaluation algorithm of the vertical acceleration response signal of the train wheelset is proposed. The simulation results show that the HHT energy entropy of 100% fastener looseness is less than 25%, 50%, and 75%, decreasing trend. The algorithm can recognize the looseness of track fastener through the experiment under different working conditions.