In order to further improve the processing performance of rotary ultrasonic machining (RUM), a novel longitudinal–torsional-coupled (LTC) vibration was applied to the RUM. An experimental study on quartz glass was performed to access the feasibility of the LTC-RUM of a brittle material. The LTC-RUM was executed through the addition of helical flutes on the tool of conventional longitudinal RUM (Con-RUM). The experimental results demonstrated that the LTC-RUM could reduce the cutting force by 55% and the edge chipping size at the hole exit by 45% on an average, compared to the Con-RUM. Moreover, the LTC-RUM could also improve the quality of the hole wall through the reduction of surface roughness, in particular, when the spindle speed was relatively low. The mechanism of superior processing performance of LTC-RUM involved the corresponding specific moving trajectory of diamond abrasives, along with higher lengths of lateral cracks produced during the abrasives indentation on the workpiece material. The higher edge chipping size at the hole entrance of LTC-RUM indicated a higher length of lateral cracks in LTC-RUM, due to the increase in the maximum cutting speed. Furthermore, the effect of spindle speed on the cutting force and surface roughness variations verified the important role of the moving trajectory of the diamond abrasive in the superior processing performance mechanism of LTC-RUM.