Using Hamilton system to analyze the nonlinear mechanical behavior of in-situ coal-rock mass has the advantage of high precision and symplectic preservation. Based on the principle of minimum potential energy, the modified H-R variational principle, combined with the symplectic space finite element-time subdomain method, the multi-layer symplectic element control equation is established. The dynamic displacement and stress parameters of each element node of layered coal-rock mass at any time are obtained by iterative calculation, which provides an efficient and good recursive algorithm for the dynamic response characteristics of layered coal-rock mass. By analyzing the torsional stress of layered coal-rock mass in the thickness direction, the principal stress of stress concentration in the horizontal direction and the diagonal shear force under the initial state of impact load loading, the weak layer position, dynamic mechanical response and stress transfer law in the thickness direction of layered coal-rock mass are determined according to the third strength criterion. Finally, the formation law of the maximum shear stress failure circle in the dynamic damage and failure process of layered coal-rock mass is discussed. The layered shear stress circle is the initial condition of damage, and the dynamic impact load is used as the excitation power to make the interlayer torsional failure. In the horizontal and thickness directions, the “O-X” failure surface with time changes will be formed along the “X” -shaped shear line, and the “V” -shaped spalling surface will be presented on the four sides.