The effects of temperature and pre-strain levels on the stress relaxation behavior and corresponding microstructural evolutions of Ti-6Al-4V alloys have been investigated experimentally and numerically in this study. A series of tests (stress relaxation (SR) and repeated stress relaxation (RSR)) and microstructural observations (scanning electron microscope) have been performed, based on which the deformation-related variables, i.e., stress component and activation energy, as a function of the testing time are calculated according to the classical thermal activation theories. The experimental SR behavior and the obtained thermal related variables show that at lower temperatures (700 °C and 750 °C), a large number of dislocations introduced by plastic loading enhance dislocation slip/climb creep, giving rise to rapid relaxation compared with those with elastic loading conditions at the same temperature. At higher temperatures (800 °C and 850 °C), a similar SR phenomenon has been observed at both elastic and plastic loading conditions, which is due to the severe interaction between diffusion creep and dislocation creep after the loading stage. Based on these results, a unified constitutive equation has been proposed to successfully predict the behavior of the whole stress relaxation process composed of the loading stage and subsequent SR stage. The model considering the continuous evolution of internal variables, e.g., dislocation density and lamellar width, in two stages can predict the stress response and microstructure variation with different temperatures from elastic to plastic loading and provide a foundation to effectively optimize the hot forming process combining pre-deformation and stress relaxation.