Axial ultrasonic vibration-assisted cutting (AUVC) has proved to have better machining performance compared with conventional cutting methods; however, the effect of numerous and complex influencing factors on machining performance has not been clearly revealed and a recommended combination of cutting conditions has not been proposed yet, especially for difficult-to-machine material such as Ti6Al4V alloy. This paper focuses on experimental and theoretical investigation into machining performance when cutting Ti6Al4V with the AUVC method. First, a retrospective of the separation characteristics of AUVC is provided and the variable parameter cutting characteristics are demonstrated. We classify the influencing factors on machining performance into four categories: machining parameters, vibration parameters, tool choice, and cooling conditions. The relationship between these factors in terms of their effect on machining performance is established theoretically. Then, it describes experiments to determine the influence of these factors on cutting force, tool life, and surface roughness. For absolute influence, the orders for cutting force, tool life, and surface roughness are respectively cutting depth > amplitude > feed rate > rotation speed, rotation speed > feed rate > amplitude > cutting depth, and feed rate > amplitude > cutting depth > rotation speed. However, for relative influence, the order is unified as: amplitude > feed rate > rotation speed > cutting depth. Finally, it suggests a smaller feed rate, larger amplitude, moderate rotation speed, and smaller cutting depth in addition to a WC tool coated with TiAlN and used under HPC cooling condition for optimal performance of AUVC. This recommendation is based on the theoretical analysis and experimental results of cutting force, surface roughness, and tool life.