The utilization of abrasive water jet (AWJ) has garnered notable attention in subsurface engineering, as well as unconventional natural gas development, geothermal energy extraction, and tunnel excavation. The efficiency of construction operations is contingent upon rock fragmentation, which is controlled by AWJ parameters and rock properties. Currently, the parameter settings for rock fragmentation by AWJ predominantly rely on empirical approaches, and existing prediction models have large errors due to a limited number of training samples. In this study, we propose a combined support vector machine (SVM) and whale optimization algorithm (WOA) model. To test the model's predictive performance for rock-breaking depth, a database consisting of eight input parameters is constructed. These parameters include AWJ pressure, target distance, lateral velocity, abrasive types, mass flow rate, abrasive particle size, rock types, and rock uniaxial compressive strength. Additionally, to demonstrate the superiority of the WOA-SVM model, three other predictive models based on the back propagation (BP) network, SVM, and Random Forest (RF) are established, compared, and evaluated. The results show that the optimized WOA-SVM model is the most accurate in predicting rock cutting depth, achieving a precision rate of 0.972 25 compared to other models (BP: 0.9536; RF: 0.9681; SVM: 0.9687). Furthermore, sensitivity analysis highlights that lateral velocity exhibits the highest impact on the model, followed by jet pressure and the uniaxial compressive strength of rock. This underscores the critical importance of prioritizing the adjustment of lateral velocity, AWJ pressure, and rock properties when engaging in rock-cutting operations.