To improve the rust removal effect of grade A marine steel by ultra-high-pressure water jets, the rust removal efficiency and rust removal quality are used as measurement standards. Taking jet pressure, standoff distance, and nozzle traverse speed as influencing factors, an orthogonal experimental table was established, and different combinations of process parameters were used to impact the surface of grade A marine steel. The rust removal efficiency is expressed by the rust removal area per unit time. The larger the rust removal area, the higher the rust removal efficiency. The rust removal quality is expressed by the erosion depth. The smaller the erosion depth, the higher the rust removal quality. Using the range analysis method to process the orthogonal experimental data, we determined the optimal process parameters for the rust removal efficiency and rust removal quality. The microscopic morphology of marine grade A steel after rust removal was observed using a confocal microscope to determine the sample with the best rust removal effect. The results indicate that the optimal process parameters for rust removal efficiency are jet pressure of 200 MPa, standoff distance of 90 mm, and nozzle traverse speed of 240 mm/min. The optimal process parameters for rust removal quality are jet pressure of 160 MPa, standoff distance of 60 mm, and nozzle traverse speed of 240 mm/min. The optimal process parameters for rust removal effect are jet pressure of 160 MPa, standoff distance of 90 mm, and nozzle traverse speed of 240 mm/min. By analyzing the microscopic morphology of the samples, the process parameters of the sample with the best rust removal effect were determined to be consistent with the optimal process parameters obtained by range analysis, verifying the accuracy of the optimal process parameters. Compared with the existing jet machining technology, the optimal process parameters obtained in this paper take into account both the rust removal quality and rust removal efficiency, and they improve the rust removal effect.