The instability of dams will bring immeasurable personal and property losses to the downstream, so it has always been a trendy topic worthy of investigation. Currently, the rigid body limit equilibrium method is the most commonly used method for the dynamic stability analysis of dams. However, under the action of earthquakes, the instability of the integral dam-foundation system threatens the safety of the dams and is of great concern. In this paper, a stability analysis method that can reflect the complex geological structural forms of dam foundations is proposed in this paper. The advantages are that this method deals with the difficulty in assuming sliding surfaces and the lack of quantitative criteria for the dynamic instability analysis of dams with complex geological structural forms of dam foundations. In addition, through the method, the sliding channels that may appear in the dam foundations can be automatically searched under random earthquake action, and the safety factors of the dynamic instability of dams be quantitatively obtained. Taking a high RCC gravity dam under construction in China as an example, the proposed method is applied to the three-dimensional finite element model of the dam-foundation system of this dam, and then the dynamic stability calculation is carried out. Through this method, the formation process of the dam foundation’s plastic zone and the failure of sliding channels with different strength reduction coefficients are studied on and analyzed detailedly, and the quantitative acquisition of the safety factors is realized. The results show that the method is reasonable and feasible, and helps provide a new idea and method for the dynamic stability analysis of dams.