The paper develops a general scheme for viscoelastic materials, where the constitutive properties are described by means of measures of strain, stress, heat flux, and their time derivatives. The constitutive functions are required to be consistent with the second law of thermodynamics. Indeed, a new view is associated with the second law: the non-negative expression of the entropy production is set equal to a further constitutive function. The introduction of the entropy production as a constitutive function allows for a much wider range of models. Within this range, a scheme to obtain nonlinear models of thermo-viscoelastic materials subject to large deformations is established. Notably, the Kelvin–Voigt, Maxwell, Burgers, and Oldroyd-B viscoelastic models, along with the Maxwell–Cattaneo heat conduction, are obtained as special cases. The scheme allows also for modelling the visco-plastic materials, such as the Prandtl–Reuss work-hardening function and the Bingham–Norton fluid.