Improving compactness is essential for high-power microwave (HPM) sources. In this paper, a novel reflective modulation cavity is proposed and investigated in a V-band relativistic coaxial transit-time oscillator (RCTTO). The cold cavity analyses and particle-in-cell simulations show that the reflective modulation cavity has larger reflection coefficients of TEM mode and stronger electron beam modulation capability when compared with a uniform modulation cavity. When the input diode voltage is 391 kV, the beam current is 4.91 kA, and when the guiding magnetic field is 0.6 T, the compact V-band RCTTO produces an output microwave power of 518 MW (conversion efficiency of 27.0%). Compared with the original RCTTO, the compact V-band RCTTO featuring a reflective modulation cavity exhibits a 24.8% increase in output power and a 5.4% improvement in efficiency, and the axial length of the magnetic field uniform region is reduced by 24.2%. The compact V-band RCTTO also demonstrates a broad operation voltage range, indicating potential for stable operation with voltage fluctuations in experiments. Furthermore, the reflective modulation cavity can be integrated into other high-frequency O-type HPM devices to enhance compactness, thereby diminishing the demands on the magnetic field region, which is advantageous for the future permanent packaging of HPM sources.