Total Routhian surface calculations have been performed to investigate rapidly rotating transfermium nuclei, the heaviest nuclei accessible by detailed spectroscopy experiments. The observed fast alignment in 252 No and slow alignment in 254 No are well reproduced by the calculations incorporating high-order deformations. The different rotational behaviors of 252 No and 254 No can be understood for the first time in terms of β 6 deformation that decreases the energies of the νj 15/2 intruder orbitals below the N = 152 gap. Our investigations reveal the importance of high-order deformation in describing not only the multiquasiparticle states but also the rotational spectra, both providing probes of the single-particle structure concerning the expected doubly magic superheavy nuclei.DOI: 10.1103/PhysRevC.86.011301 PACS number(s): 21.10. Re, 21.60.Cs, 23.20.Lv, 27.90.+b Together with the exploration of nuclei far away from the stability line using radioactive nuclear beams, the synthesis of superheavy nuclei towards the predicted "island of stability" by fusion reactions is the focus of current research on atomic nuclei [1,2]. The occurrence of superheavy nuclei is due to the quantum shell effect (see, e.g., Ref. [3]) that overcomes the strong Coulomb repulsion between the large number of protons. The shell effect peaks at the expected doubly magic nucleus next after 208 Pb, the center of the stability island. Unfortunately, various theories give rise to different magic numbers, and available experiments have not been able to confirm or exclude any of them. Nevertheless, one can obtain single-particle information that is intimately related to the shell structure of superheavy nuclei from transfermium nuclei where γ -ray spectroscopy has been accessible for experiments [4,5].Transfermium nuclei have been found to be deformed. For example, β 2 ≈ 0.27 has been derived for 254 No from the measured ground-state band [6]. The observation of K isomers with highly hindered decays in 254 No [7][8][9] points to an axially symmetric shape for the nucleus. The deformation can bring the single-particle levels from the next shell across the predicted closure down to the Fermi surface. They play an active role in both nuclear noncollective and collective motions that in turn can serve as probes of the single-particle structure. For example, the observed K π = 3 + state formed by broken-pair excitation in 254 No is of special interest [7]. This is because the π 1/2 − [521] orbital occupied by one unpaired nucleon stems from the spherical orbital 2f 5/2 whose position relative to the spin-orbit partner 2f 7/2 determines whether Z = 114 is a magic number for the "island of stability." On the other hand, high-j intruder orbitals sensitively respond to the Coriolis force during collective rotation. The observation of upbending or backbending phenomena is usually associated with the alignment of high-j intruder orbitals. The spectroscopy experiments on transfermium nuclei provide a * hlliu@mail.xjtu.edu.cn testing ground for...