The retained coal in the end slope of an open-pit mine can be mined by the highwall mining techniques. However, the instability mechanism of the reserved rib pillar under dynamic loads of mining haul trucks and static loads of the overlying strata is not clear, which restricts the safe and efficient application of highwall mining. In this study, the load-bearing model of the rib pillar in highwall mining was established, the cusp catastrophe theory and the safety coefficient of the rib pillar were considered, and the criterion equations of the rib pillar stability were proposed. Based on the limit equilibrium theory, the limit stress of the rib pillar was analyzed, and the calculation equations of plastic zone width of the rib pillar in highwall mining were obtained. Based on the Winkler foundation beam theory, the elastic foundation beam model composed of the rib pillar and roof under the highwall mining was established, and the calculation equations for the compression of the rib pillar under dynamic and static loads were developed. The results showed that with the increase of the rib pillar width, the total compression of the rib pillar under dynamic and static loads decreases nonlinearly, and the compression of the rib pillar caused by static loads of the overlying strata and trucks has a decisive role. Numerical simulation and theoretical calculation were also performed in this study. In the numerical simulation, the coal seam with a buried depth of 122 m and a thickness of 3 m is mined by highwall mining techniques. According to the established rib pillar instability model of the highwall mining system, it is found that when the mining opening width is 3 m, the reasonable width of the rib pillar is at least 1.3 m, and the safety factor of the rib pillar is 1.3. The numerical simulation results are in good agreement with the results of theoretical calculation, which verifies the feasibility of the theoretical analysis of the rib pillar stability. This research provides a reference for the stability analysis of rib pillars under highwall mining.