We derive analytic expressions for the three-dimensional coherent transfer function (CTF) and coherent spread function (CSF) for coherent holographic image reconstruction by phase transfer (CHIRPT) microscopy with monochromatic and broadband illumination sources. The 3D CSF and CTF were used to simulate CHIRPT images, and the results show excellent agreement with experimental data. Finally, we show that the formalism presented here for computing the CSF/CTF pair in CHIRPT microscopy can be readily extended to other forms of single-pixel imaging, such as spatial-frequency-modulated imaging.