In modern endodontics, nickel-titanium (NiTi) rotary instruments are used on a large scale for root canal shaping. Nevertheless, the separation of an instrument is a serious concern during shaping. The aim of this study is to determine and compare the torsional fracture characteristics of three types of NiTi endodontic instruments, each with different cross-section designs and movements performed during root canal shaping: Endostar E3 (Endostar, Poldent Co. Ltd., Warsaw, Poland); Reciproc R25 (VDW, Munich, Germany); and Protaper Next X2 (Dentsply Maillefer, Ballaigues, Switzerland). Fifteen instruments are used in this study, divided in three groups (n = 5): Group Endostar, Group Reciproc and Group Protaper. For testing, each instrument is used to shape five simulated root canals, following which its torsional stress to failure is measured. The fracture lengths of all three groups are roughly between 2 and 3 mm from the tip. Higher values of the moment of torsion in fracture, and smaller values of the maximum twisting angle are observed for Group Endostar, as well as closer to circular cross-sections. However, the values of the shear tension are similar for all three groups, because the disadvantage given by the fracture section shape for Groups Reciproc and Protaper is compensated either by size or by intrinsic properties of the instrument material. For the shear tension the Endostar values are insignificantly increased (Kruskal–Wallis test, p = 0.207), and in the case of the maximum twist angle the Protaper values are insignificantly increased (Kruskal–Wallis test, p = 0.287). Because of the instruments shape and conicity, the analysis had to be carried out separately with regard to the length of the fractured tip. Rules-of-thumb are extracted from the study for current practice: if a blockage of the first 2 to 3 mm part of the tip can be anticipated (by the excessive curving of the instrument), the handpiece must be adjusted to torque values that do not exceed 1.5 to 2.5 N · cm for Endostar and 1 to 2 N · cm for Reciproc and Protaper instruments.