A common way of performing phase-shift-based time-of-flight imaging combines the emission of a continuous-wave (CW) illumination signal with correlation with some reference signals at the detector array. This is the case for the well-known photonic mixer device (PMD), which correlates against displaced versions of the illumination control signal, at known phase shifts, and requires only four correlation values to estimate the phase shift. The main drawback of such approaches is that they require the assumption of nonrealistic hypothesis regarding the sensing process, leading to simple sensing models that, despite allowing fast depth estimation from few acquisitions, often ignore relevant considerations for real operation, leaving the door open for systematic errors that affect the final depth accuracy. Typical examples are ignoring the effect of the illumination devices on the final shape of the illumination signal, supposing a sinusoidal reference signal at pixel level, or not accounting for multipath effects. In this work, we present a novel framework for PMD-based signal acquisition and recovery that exploits the sparsity of CW illumination signals in the frequency domain to provide accurate reconstruction of the illumination waveforms as received by the PMD pixels. Our method is extremely robust to signal distortion and noise, since no assumption is made on the illumination signal, other than being a periodic signal. Our approach ensures that no valuable information is lost during the sensing process and allows, therefore, accurate phase shift estimation in a wider range of operation conditions, getting rid of unrealistic assumptions.