The highwall miner can be used to mine the retained coal in the end slope of an open-pit mine. However, the instability mechanism of the reserved rib pillar under dynamic and static loads is not clear, which restricts the safe and efficient application of the highwall mining system. In this study, the load-bearing model of the rib pillar in highwall mining was established, the cusp catastrophe theory and the safety coefficient of the rib pillar were considered, and the criterion equations of the rib pillar stability were proposed. Based on the limit equilibrium theory, the limit stress of the rib pillar was analyzed, and the calculation equations of plastic zone width of the rib pillar in highwall mining were obtained. Based on the Winkler foundation beam theory, the elastic foundation beam model composed of the rib pillar and roof under the highwall mining was established, and the calculation equations for the compression of the rib pillar under dynamic and static loads were developed. The results show that with the increase of the rib pillar width, the total compression of the rib pillar under dynamic and static loads approximately decreases in an inverse function, and the compression of the rib pillar caused by static loads of the overlying strata and trucks has a decisive role. Numerical simulation and theoretical calculation were performed in this study. In the Numerical simulation, the coal seam with a buried depth of 122 m and a thickness of 3 m was mined by the highwall miner. According to the established rib pillar instability model of the highwall mining system, it is found that when the mining tunnel width is 3 m, the reasonable width of the rib pillar is at least 1.3 m, and the safety factor of the rib pillar is 1.3. The numerical simulation results are in good agreement with the results of theoretical calculation, which verifies the feasibility of the theoretical analysis of the rib pillar stability. The research results can provide an important reference for the stability analysis of rib pillars under highwall mining.