The precision measurement on surperfinish surface scratch has been currently paid much attention to on electronic products. To meet these demands, a novel method has been proposed which is based on heterodyne interferometry that utilizes birefringent lens as beam splitter and cantilever tip as scanning probe to get the measurement values of sample topography. But the optical nonlinear errors affect the measurement precision of the system. In this paper, we adopt the Jones matrix to analyze the elliptic polarization caused by the three factors existing at the same time, which are polarization ellipticity of laser source, installation orientation error and phase retardation of birefringent lens. The measurement errors of frequency mixing about these elliptic polarization beams arriving at the photodiode detecter are studied by vector theory. The results show that the measurement errors are periodic errors, and they will change from 1.24nm to 3.94 nm when the magnitude of orientation error of birefringent lens changes from 1°to 5°. Also, the methods of reducing measurement errors according to the numerical results in the system are suggested. The measurement precision will be improved by reducing the orientation error or choosing high performance laser source.