The nitrogen-vacancy (NV) center in diamond is well known in quantum metrology and quantum information for its favorable spin and optical properties, which span a wide temperature range from near zero to over 600 K. Despite its prominence, the NV center's photophysics is incompletely understood, especially at intermediate temperatures between 10-100 K where phonons become activated. In this paper, we present a rate model able to describe the crossover from the low-temperature to the high-temperature regime. Key to the model is a phonondriven hopping between the two orbital branches in the excited state (ES), which accelerates spin relaxation via an interplay with the ES spin precession. We extend our model to include magnetic and electric fields as well as crystal strain, allowing us to simulate the population dynamics over a wide range of experimental conditions. Our model recovers existing descriptions for the low-and high-temperature limits and successfully explains various sets of literature data. Further, the model allows us to predict experimental observables, in particular the photoluminescence (PL) emission rate, spin contrast, and spin initialization fidelity relevant for quantum applications. Lastly, our model allows probing the electron-phonon interaction of the NV center and reveals a gap between the current understanding and recent experimental findings.