Wellbore stability problems increase with the exploration and development of oil and gas reservoirs. A new 3D non-linear failure criterion is proposed as a trigonometric function considering the intermediate principal stress (2) on the triaxial compression test data. Mohr-Coulomb and Hoek-Brown are well-known failure criteria, but they do not consider the influence of (2) on rock strength. This new criterion produces a concave surface on the principal stress space (1,2, 3) with the influence of intermediate principal stress. In this study, sensitivity analysis for the variable is also done to understand the significant influence of parameters on the accuracy of the proposed criterion. Further validation of this non-linear criterion on three principal stresses (1,2, 3) was done compared with linear regression and second-degree polynomial regression results. It has been observed that the new non-linear 3D criterion with five material parameters reveals a good fit compared to linear regression and second-degree polynomial regression, which have four and six material parameters, respectively. The new non-linear criterion was further validated by comparison with existing criteria like the Priest, Drucker-Prager, and Mogi-Coulomb. It has been observed that the new 3D non-linear criterion shows a more accurate result than these existing criteria as certain rock types exhibit coefficient of determination (DC) values near one, precisely 0.95 for inada granite, 0.94 for orikabe monzonite, and 0.91 for KTB amphibolite. In contrast, other rock types have DC values ranging from 0.7 to 0.9. The new 3D non-linear criterion also yields lower root means square error (RMSE) values than the Mogi-Coulomb criterion for seven rock types. Specifically, the RMSE values by the new criterion are as follows: KTB amphibolite - 40.03 MPa, Dunham dolomite - 15.16 MPa, Shirahama sandstone - 9.08 MPa, Manazuru andesite - 22.14 MPa, Inada granite - 35.47 MPa, and Coconino sandstone - 19.047 MPa. This new 3D criterion gave precise predictions of the failure of the formation under in-situ stresses and was further helpful for the simulation of the wellbore in the petroleum industry. The variable in the new 3D criterion should be calculated from triaxial compression test data for each formation rock before applying this criterion to the wellbore stability problem and the sand production problem.