Determination of rock mass strength is important in most rock engineering analyses that are based on solid mechanics. Better understanding of rock mass strength will facilitate proper engineering design of structures. The loading condition of rock is complex, and a rock failure criterion represents the strength of rock under different loading conditions. Many failure criteria have been proposed by researchers during the past decades. In rock engineering practice, the linear Mohr-Coulomb (MC) criterion and the nonlinear Hoek-Brown (HB) criterion are widely used in conditions where 2 = 3 , largely due to their simplicity in formulation and the large amount of experimental data available (Melkoumian, Priest, and Hunt, 2009), despite the fact that the physical sense of the former is obscure and the latter is of a purely empirical character. Both the abovementioned criteria suffer limitations arising from ignoring the effect of the intermediate principal stress ( 2 ) on rock strength (Chang and Haimson, 2012;Haimson, 2006;Tiwari and Rao, 2004). Much of the experimental evidence accumulated so far strongly suggests that 2 has a considerable effect on rock strength (Chang and Haimson, 2012;Tanapol, Chaowarin, and Kittitep, 2013;Tarasov and Potvin, 2013). To incorporate the influence of 2 , several general failure criteria have been proposed, among which the Drucker-Prager criterion, the modified Lade criterion, the 3D Hoek-Brown criterion, and the unified strength criterion are well-known (Xie and Chen, 2004). However, these failure criteria are not commonly employed in practice. According to several comparative studies, none of the existing 3D failure criteria has a significant advantage over others, from both mathematical and practical points of view.A perceived shortcoming of the traditional failure criteria is that they are established only on the basis of macroscopic experiments and combined theoretical analysis, and they do not attempt to microscopically analyse the failure mechanism and failure criterion of rock under different loading conditions. In fact, the study of deformation and failure of rock on the micro-meso scale can reveal the innate character and relationship between the macroscopic response, in the form of deformation or failure, and the intrinsic microscopic mechanism (Adelinet et al., 2013;Lin et al., 2015; Zhou and Linn, 2013; Zhou and Linn, 2014).An investigation of failure modes and failure criteria of rock in complex stress states by Z. Li*, J. Shi*, and A. Tang* Rock in engineering and geological environments is usually in complex stress states. Based on many experimental results for rock under different loading conditions, and combined with failure modes found in previous studies, stress triaxiality is introduced to describe different stress states, and the relationship between failure modes and stress triaxiality is analysed in detail. For a given kind of rock, with decreasing stress triaxiality the failure mechanism changes from tension fracture to local shear failure and general shear f...