A novel heterodyne grating interferometer based on a quasi-common-optical-path design is proposed for displacement measurement. The quasi-common-optical-path design relies on the phase shift between the zeroth and first diffraction grating orders which have been rotated in polarization using a half-wave plate. We achieved a measurement resolution better than 3 nm with a system stability of less than 14 nm over 1 h. We discussed the performances of the system addressing the effect of dominant errors, namely grating pitch, frequency mixing, polarization mixing and polarization-frequency mixing. While the theoretically quasi-common-optical-path heterodyne grating system allows for sub-nanometer resolution, we found that the measurement resolution here is limited by the displacement stage. Relying on heterodyne interferometric phase measurement combined with quasi-common-optical configuration, the proposed and demonstrated method has the advantages of high measurement resolution, relatively straightforward operation and high stability.