A subnanometer resolution displacement sensor based on a grating interferometric cavity with intensity compensation and phase modulation is proposed and experimentally demonstrated in this paper. The grating interferometric cavity is composed of a frequency-stabilized laser source, a diffraction grating, and a mirror. To realize a subnanometer resolution, the intensity compensation and phase modulation technique are introduced, which are achieved by an intensity compensation light path, three closed placed photodetectors, a processing circuit and a piezoelectric ceramic transducer, and a lock-in amplifier. The intensity compensation technique can improve the stability of the output intensity signal greatly while the phase modulation technique can increase the signal-tonoise ratio dramatically. The detected signal is intensity modulated and processed by a particular arithmetic circuit. Experimental results indicate that the sensitivity of this displacement sensor is 44.75 mV/nm and the highest resolution can reach 0.017 nm, which is 27 times better than the one without intensity compensation and phase modulation. As a high-performance sensor with immunity to electromagnetic interference, this displacement sensor has potential to be used in nanoscience and technology.