Shear failure often occurs in engineering rock mass (such as inclined pillar) in gently inclined strata. Prediction and characterization the orientation of shear failure plane is the foundation of rock mass engineering reinforcement. In this paper, sandstone samples are used to perform uniaxial and shear tests to obtain the basic mechanical parameters. Then, by employing the numerical method, the combined compression–shear loading tests were carried out for inclined specimens varied from 0° to 25° at an interval of 5°, to obtain the dip effect on the orientation of rock failure plane. The results show that the failure plane of rock changes with the change of dip angle of rock sample. Based on the Mohr–Coulomb criterion, the ultimate stress state of rock was characterized under combined compression–shear loading. The ultimate strength of rock is equal to the ratio of the stress circle radius of rock under combined compression–shear condition to the stress circle radius of rock under uniaxial compression condition, multiplied by the uniaxial compressive strength. The fracture angle of rock was defined under combined compression–shear loading. A theoretical model was developed for predicting the fracture angle. The developed model could be characterized by internal friction angle, dip angle of rock sample and Poisson's ratio. Finally, the numerical results of the fracture angle were analyzed, which are consistent with the predicted results of the model. The investigation shows that the rock fracture angle has a dip effect, which decreases with the increase of the inclination angle of the sample. The research results provide a new means to identify the potential failure plane of engineering rock mass, and lay a theoretical foundation for calculating the orientation of rock fracture plane.