Micro absolute distance measurement (MADM) is widely used in industrial and military fields. To achieve high accuracy and frequency response, a polarized low-coherence interferometry (PLCI)-based method for MADM is proposed. The nearly linear relationship between the envelope center and m-order PLCI fringe (PLCIF) peak center is found and verified. Dispersion compensation is achieved by fringe peak position estimation and polynomial fitting to get rid of the dependence on an a priori model and birefringence parameters, and make this method very robust. Meanwhile, the zero-order PLCIF center is estimated and located to demodulate the measured displacement. Then, the measurement accuracy is raised by polynomial fittings. In comparison to conventional methods, the proposed method can effectively avoid jump errors and has a higher accuracy. Experimental results indicate that the measurement accuracy is higher than 19.51 nm, the resolution is better than 2 nm, and its processing data rate can reach 35 kHz.