The potential of microwave Doppler radar in non-contact vital sign detection is significant; however, prevailing radar-based heart rate (HR) and heart rate variability (HRV) monitoring technologies often necessitate data lengths surpassing 10 s, leading to increased detection latency and inaccurate HRV estimates. To address this problem, this paper introduces a novel network integrating a frequency representation module and a residual in residual module for the precise estimation and tracking of HR from concise time series, followed by HRV monitoring. The network adeptly transforms radar signals from the time domain to the frequency domain, yielding high-resolution spectrum representation within specified frequency intervals. This significantly reduces latency and improves HRV estimation accuracy by using data that are only 4 s in length. This study uses simulation data, Frequency-Modulated Continuous-Wave radar-measured data, and Continuous-Wave radar data to validate the model. Experimental results show that despite the shortened data length, the average heart rate measurement accuracy of the algorithm remains above 95% with no loss of estimation accuracy. This study contributes an efficient heart rate variability estimation algorithm to the domain of non-contact vital sign detection, offering significant practical application value.