PurposeIn this paper, the authors aim to propose an effective method to indirectly determine nonlinear elastic shear stress-strain constitutive relationships for nonlinear elasticity materials, and then study the nonlinear free torsional vibration of Al–1%Si shaft.Design/methodology/approachIn this study the authors use BoxLucas1 model to fit the determined-experimentally nonlinear elastic normal stress–strain constitutive relationship curve of Al–1%Si, a typical case of isotropic nonlinear elasticity materials, and then derive its nonlinear shear stress-strain constitutive relationships based on the fitting constitutive relationships and general equations of plane-stress and plane-strain transformation. Hamilton’s principle is utilized to gain nonlinear governing equation and boundary conditions for free torsional vibration of Al–1%Si shaft. Differential quadrature method and an iterative algorithm are employed to numerically solve the gained equations of motion.Findings The effect of four variables, namely dimensionless fundamental vibration amplitude ϑmax, radius α and length β, and nonlinear-elasticity intensity factor δ, on frequencies and mode shapes of the shafts is obtained. Numerical results are in good agreement with reference solutions, and show that compared with linearly elastic shear stress-strain constitutive relationships of the shafts made of the nonlinear elasticity materials, its actual nonlinearly elastic shear stress-strain constitutive relationships have smaller torsion frequencies. In addition, but β having opposite hardening effect, the rest of the four variables have softening effect on nonlinearly elastic torsion frequencies. Eventually, taking into account nonlinearly elastic shear stress-strain constitutive relationships, changes of the four factors, i.e. ϑmax, α, β and δ, cause inflation and deflation behaviors of mode shapes in nonlinear free torsional vibration.Originality/valueThe study could provide a reference for indirectly determining nonlinear elastic shear stress-strain constitutive relationships for nonlinear elasticity materials and for structure design of torsional shaft made of nonlinear elasticity materials.