A rock mass in-situ rotary cutting test system (RMIRCS) was developed to estimate the mechanical properties (i.e. compressive strength and elastic modulus) of rock masses. The pressure-on-bit and torque-on-bit of the RMIRCS were investigated in detail over a wide range of revolution speed and penetration rate. A simple experimental model is proposed to describe the qualitative relationships between rotary cutting test data and mechanical parameters. The existence of a linear constraint between pressure-on-bit and torque-on-bit and the depth of rotary cutting exhibits good agreement with the proposed model. The mechanical properties obtained from the quasi-static compression tests were in agreement with those of the RMIRCS at lower revolutions rate of 300 rpm and 350 rpm, which suggests that the developed RMIRCS is valid. Furthermore, the experimental test results on rock mass demonstrated that the mechanical properties of rock masses are revolutions speed-dependent and drilling rate-dependent.