The amplitude and phase of a material’s nonlinear optical response provide insight into the underlying electronic dynamics that determine its optical properties. Phase-sensitive nonlinear spectroscopy techniques are widely implemented to explore these dynamics through demodulation of the complex optical signal field into its quadrature components; however, complete reconstruction of the optical response requires measuring both the amplitude and phase of each quadrature, which is often lost in standard detection methods. Here, we implement a heterodyne-detection scheme to fully reconstruct the amplitude and phase response of spectral hole-burning from InAs/GaAs charged quantum dots. We observe an ultra-narrow absorption profile and a corresponding dispersive lineshape of the phase, which reflect the nanosecond optical coherence time of the charged exciton transition. Simultaneously, the measurements are sensitive to electron spin relaxation dynamics on a millisecond timescale, as this manifests as a magnetic-field dependent delay of the amplitude and phase modulation. Appreciable amplitude modulation depth and nonlinear phase shift up to ~0.09×π radians (16°) are demonstrated, providing new possibilities for quadrature modulation at faint photon levels with several independent control parameters, including photon number, modulation frequency, detuning, and externally applied fields.