The effect of phase jitter on differential sampling using the programmable Josephson voltage standard (PJVS) system is studied in this paper. A phase jitter model is established for the measured signal, and compensation coefficients for phase jitter removal are derived for three different post-processing methods based on the discrete Fourier transform algorithm (DFT). Based on our analysis, the phase jitter compensation coefficients are determined by the phase jitter angle distribution and harmonic order. Furthermore, after analyzing and simulating various common distributions, the phase jitter compensation coefficients have been verified. The simulation shows that when the standard deviation of the phase jitter angle is 20 ns, and the frequency of the measuring waveform is 3.46 kHz, the influence of the phase jitter is 1 × 10−7. The results of the simulation indicate that, in the differential sampling of AC waveforms using a PJVS system, phase jitter is one of the error terms for an uncertainty budget that cannot be neglected, particularly as the frequency of the measured waveforms increases.