Steep surrounding rock significantly challenges tunnel stability by affecting the stress distribution and deformation behavior. The angle of dip in surrounding rock greatly influences these factors, heightening the risk of instability along bedding planes, particularly under high ground stress conditions. This paper presents a comprehensive analysis of steep rock strata mechanical properties based on a railway tunnel in Yunnan Province, China. It incorporates long-term field monitoring and various laboratory tests, including point load, triaxial, and loose circle tests. Using experimental data, this study simulated the failure processes of steep surrounding rock and tunnel structures with a custom finite element method (FEM) integrated with the volume of fluid (VOF) approach. The analysis summarized the deformation patterns, investigated the causes of inverted arch deformation and failure, and proposed countermeasures. The findings reveal that increasing the rock dip angle results in greater deformation and accelerated failure rates, with the surrounding rock’s loose zone stabilizing at approximately 8 m once deformation stabilizes. At a surface deformation of 8 cm, the failure zone extends to 6 m; however, this extension occurs more rapidly with higher lateral pressure coefficients. Additionally, failure zones develop more quickly in thin, soft rock on steep slopes compared to uniform rock formations. The rise of the tunnel floor is attributed to the steeply inclined, thin surrounding rock. To enhance bottom structure stiffness, this study recommends incorporating an inverted arch structure and increasing both the number and strength of the anchor bolts.