Ultra-slow relaxation process of polymers has the memory-dependent feature, integer-order thermoviscoelastic models may fail to describe the dynamic behaviors of viscoelastic structures accurately. Additionally, it is noticed that the small-scale effect of elastic deformation and heat conduction in a non-isothermal temperature environment is becoming significant due to the development of micro-devices. To better capture the memory-dependent effect and the small-scale effect of viscoelastic micro-structures in heat transfer environment, as a first attempt, present work focuses on developing a refined fractional Kelvin-Voigt thermoviscoelastic model by incorporating the nonlocal dual-phase-lag (NDPL) heat conduction model and the modified coupled stress theory (MCST). Then, the model is applied to investigating the transient response of a polymer microbeam subjected to a harmonic thermal loading. The governing equations involving the modified parameters are formulated and then solved by Laplace transform method. Some parametric results are demonstrated to display the impacts of the nonlocal thermal parameter, the material length-scale parameter and the fractional-order parameter on the considered physical quantities. The results show that the small-scale effect and the memory-dependent effects strongly depend on the polymer micro-structure characteristics in thermal environment.