We theorize the surface step characterization by reflected incoherent-light differential interference microscopy with consideration of the optical diffraction effect. With the integration of localization analysis, we develop a quantitative differential interference optical system, by which we demonstrate that the axial resolution of measuring surface height variation is sensitive to the shear distance between the two spatially differentiated beams. We fabricate three nanometer-size steps by√ photolithography, and successfully characterize their 1D height variations with 0.13 nm/ Hz axial precision. Our result suggests that the optical differential interference microscopy can be used for real-time characterization of surface structure with a subnanometer accuracy and a large field of view, which is greatly beneficial to the surface characterization of micro/nano-electromechanical systems.