Excavation unloading significantly contributes to rock slope failure in an open-pit mine. At present, there is no relevant theoretical study on the failure mechanism of the rock slope under excavation unloading. Therefore, in this study, based on the theory of fracture mechanics, the expression of the stress intensity factor at the crack tip on the rock mass at the vertical distance, h, from the slope top under excavation unloading is derived, the calculation method of the crack initiation angle is given, the expression of the ultimate safe height of the slope under unloading is obtained, and the ratio of the fracture toughness of the slope rock mass to the composite stress intensity factor at the crack tip on the rock mass is defined as the slope stability factor, which is verified by an engineering example. The results show that the crack initiation angle decreased when crack inclination was increased, and the crack initiation angle increased when the side-pressure coefficient, slope angle, and friction coefficient were increased. The ultimate safety height of the slope decreased first and then increased with the increase in the crack angle, and it was approximately linear with the crack length and inversely proportional with the slope angle. The stability coefficient calculated by this method is the same as that calculated by the limit equilibrium method and is small, which indicates the accuracy and rationality of this method. Results in this study can provide a theoretical basis for understanding and controlling the slope collapse disaster induced by excavation unloading.