Physics-guided machine learning (PGML) methods are emerging as valuable tools for modelling the constitutive relations of solids due to their ability to integrate both data and physical knowledge. While various PGML approaches have successfully modeled time-independent elasticity and plasticity, viscoelasticity remains less addressed due to its dependence on both time and loading paths. Moreover, many existing methods require large datasets from experiments or physics-based simulations to effectively predict constitutive relations, and they may struggle to model viscoelasticity accurately when experimental data are scarce. This paper aims to develop a physics-guided recurrent neural network (RNN) model to predict the viscoelastic behavior of solids at large deformations with limited experimental data. The proposed model, based on a combination of gated recurrent units (GRU) and feedforward neural networks (FNN), utilizes both time and stretch (or strain) sequences as inputs, allowing it to predict stress dependent on time and loading paths. Additionally, the paper introduces a physics-guided initialization approach for GRU–FNN parameters, using numerical stress–stretch data from the generalized Maxwell model for viscoelastic VHB polymers. This initialization is performed prior to training with experimental data, helping to overcome challenges associated with data scarcity.