The use of advanced brittle composites in engineering systems has necessitated robotic rotary ultrasonic machining to attain high precision with minimal machining defects such as delamination, burrs, and cracks. Longitudinal–torsional coupled (LTC) vibrations are created by introducing helical slots to a horn’s profile to enhance the quality of ultrasonic machining. In this investigative research, modified ultrasonic horns were designed for a giant magnetostrictive transducer by generating helical slots in catenoidal and cubic polynomial profiles to attain a high amplitude ratio (TA/LA) and low stress concentrations. Novel ultrasonic horns with a giant magnetostrictive transducer were modelled to compute impedances and harmonic excitation responses. A structural dynamic analysis was conducted to investigate the effect of the location, width, depth and angle of helical slots on the Eigenfrequencies, torsional vibration amplitude, longitudinal vibration amplitude, stresses and amplitude ratio in novel LTC ultrasonic horns for different materials using the finite element method (FEM) based on the block Lanczos and full-solution methods. The newly designed horns achieved a higher amplitude ratio and lower stresses in comparison to the Bezier and industrial stepped LTC horns with the same length, end diameters and operating conditions. The novel cubic polynomial LTC ultrasonic horn was found superior to its catenoidal counterpart as a result of an 8.45% higher amplitude ratio. However, the catenoidal LTC ultrasonic horn exhibited 1.87% lower stress levels. The position of the helical slots was found to have the most significant influence on the vibration characteristics of LTC ultrasonic horns followed by the width, depth and angle. This high amplitude ratio will contribute to the improved vibration characteristics that will help realize good surface morphology when machining advanced materials.