The bearing stratum of high-rise and ultra-high-rise buildings in southwest China has inevitably faced moderately weathered red mudstone. It was a waste of the potential bearing stratum calculated according to the specification, as the bearing stratum obtained from laboratory and in situ tests was much higher than the values suggested by the specification. Rock mass surface deformation detection is of great significance in the safety management of a foundation project. Some correlation between surface deformation and failure characteristics may exist that could help to understand the bearing stratum of the moderately weathered red mudstone. This research was conducted to study the progressive failure characteristics of the moderately weathered red mudstone through surface deformation. In situ load, triaxial, and binocular visual technology were employed for data acquisition. The proposed conjecture was illustrated and verified by a group of experiments from three construction sites. Five stages could be described as the progressive failure of the moderately weathered red mudstone: compaction, elasticity, elastoplasticity, plasticity, and failure. Furthermore, the surface displacement increment fluctuates with the loading time and fades into the distance. Therefore, this research could provide a robust, practical application for analyzing the progressive failure of moderately weathered red mudstone.