The aim of this study was to investigate the shaping ability and cyclic fatigue resistance of Genius (GN) and WaveOne (WO) nickel-titanium (NiTi) instruments. Forty mesial root canals of mandibular molars were randomly divided into two groups, according to the instrument system used for root canal preparation (n = 20): GN and WO. Root canals were prepared to the full working length using the crown-down technique, with up to 40.04 instruments for GN group and 40.08 instruments for WO group. Specimens were scanned with a µCT before and after instrumentation. The changes in 3D parameters and degree of canal transportation were evaluated in the apical, middle and coronal thirds of root canals. Size 25, taper 0.04 (GN) and 0.08 (WO) instruments (n = 20) were used in simulated root canals until fracturing, and the number of cycles to failure (NCF) was recorded. The fractured files were investigated under a scanning electron microscope to characterize fractured surfaces. Data were analyzed using ANOVA post hoc Tuckey’s tests with the significance level set at 5%. The GN system was able to prepare the higher percentage of the canal surface with less canal transportation in all root canal thirds and displayed the higher resistance to cyclic fatigue in comparison to WO instruments (p < 0.05). However, the length of the fractured segments was not significantly different among the groups; meanwhile, SEM investigations revealed the typical topographic characteristics of fractured surfaces. Hybrid GN instruments demonstrated superior shaping ability and resistance to cyclic fatigue. However, both NiTi instrument systems prepared the root canals with no significant shaping errors, considerable changes in 3D parameters and no significant differences in the topographic appearances of the instrument surfaces after fractures.