The creation and evolution of nonequilibrium phonons is central in applications ranging from cosmological particle searches to decoherence processes in qubits. However, fundamental understanding of decoherence pathways for athemal phonon distributions in solid-state systems remains an open question. Using first-principles calculations, we investigate the primary decay channels of athermal phonons in two technologically-relevant semiconductors -Si and GaAs. We quantify the contributions of anharmonic, isotopic and interfacial scattering in these materials. From this, we construct a model to estimate the thermal power in a readout scheme as a functional of time. We discuss the implication of our results on noise limitations in current phonon sensor designs, and strategies for improving coherence in next-generation phonon sensors.