Linear instability analysis of a viscous swirling liquid jet surrounded by ambient gas is carried out by considering the significant influence of axial shear effect. The jet azimuthal flow is assumed as a Rankine vortex, and the non-uniform velocity distribution in the jet axial direction is approximated by parabolic and error functions. The enhancement of jet rotation is found to promote the predominant mode with larger azimuthal wavenumbers, and the mode transition is decided by the competition between centrifugal force and axial shear stress. Subsequently, the influence of the axial shear effect is examined through changing the degree of shear stress and the thickness of the gas velocity boundary layer. It is found that an increase of jet average velocity or surface velocity in the axial direction leads to the predominant mode transition to smaller azimuthal wavenumbers, due to the combined effects of shear stress and gas pressure perturbation. A larger velocity difference between ambient gas and liquid jet also promotes the predominant modes with smaller azimuthal wavenumbers, and the physical mechanism is attributed to gas pressure perturbation. Phase diagrams of different azimuthal modes are given and compared with the study of Kubitschek & Weidman (J. Fluid Mech., vol. 572, 2007, pp. 261–286), where a static swirling column without axial shear stress was considered. The strengthened axial shear effect is found to delay the transition of predominant modes with the increase of angular velocity. Experimental studies considering the swirling jets with different axial velocities are further carried out, which validate the theoretical findings. Different instability mechanisms and their transition rules are also identified through energy budget analysis. This study is expected to give scientific guidance on understanding the instability mechanisms of the swirling jets that widely exist in natural phenomena and engineering applications.