Radial jet drilling (RJD) technology has been proved to be an economical and efficient stimulation technology for oil and gas, geothermal, hydrate, etc. but conventional RJD technology adopts pure water jet to break rock and form laterals, which has low rock breaking efficiency and is unable to effectively break hard rock such as shale. Swirling abrasive jet is proposed to promote the development of RJD. Here, the characteristics of the flow field of the swirling abrasive jet nozzle and the influence of the key impeller parameters are studied by numerical simulation. The distribution and development of axial velocity, tangential velocity, and radial velocity of water and abrasive are analyzed. The results show that the swirling abrasive jet has no constant velocity core, has stronger diffusivity, and can form a larger impact area than the direct jet. Abrasive particles and water can acquire large tangential and radial velocity which can break rock under the action of shear and tensile stress efficiently. With the increase of the spinning angle, the axial velocity of the fluid decreases, and the tangential velocity increases gradually. With the increase of blade thickness, the axial velocity decreases, and the tangential velocity increases. With the increase of the number of blades, the axial velocity decreases, and the tangential velocity increases. The spinning direction almost has no effect on the flow field. Therefore, the spinning angle is recommended to be no less than 270°, blade thickness is 2.5 mm, and number of blades are 3. The research results provide theoretical guidance for the structural design of swirling abrasive jet nozzles.