According to the principle of phase-shifting interferometry and spiral phase characteristics of the vortex beam, this article proposes a method for detecting the surface profile of a transparent object, in which the +1 order vortex beam is generated by a spatial light modulator and is taken as the reference light. The influence of the nonlinear phase modulation characteristics of the spatial light modulator on the measurement precision is studied. The results show that nonlinear phase modulation has a great impact on the measurement. Then, the vortex lights with initial phases of 0, π/2, π, and 3π/2 are used to measure the H-type thin film sample based on the Twyman-Green interference system after correcting the nonlinear phase modulation characteristics. The experimental results show that the measurement error of the surface profile to an object with the theoretical value of 20 nm is 1.146 nm, and the feasibility of the optical vortex phase-shifting technique used to measure the surface profile of an object is verified.