The effective dynamic properties of specific periodic structures involving rubber-like materials can be adjusted by pre-strain, thus facilitating the design of custom acoustic filters. While nonlinear viscoelastic behaviour is one of the main features of soft solids, it has been rarely incorporated in the study of such phononic media. Here, we study the dynamic response of nonlinear viscoelastic solids within a 'small-on-large' acoustoelasticity framework, that is we consider the propagation of small amplitude waves superimposed on a large static deformation. Incompressible soft solids whose behaviour is described by the Fung-Simo quasi-linear viscoelasticy theory (QLV) are considered. We derive the incremental equations using stress-like memory variables governed by linear evolution equations. Thus, we show that wave dispersion is governed by a strain-dependent generalised Maxwell rheology. Illustrations cover the propagation of plane waves under homogeneous tensile strain in a QLV Mooney-Rivlin solid. The acoustoelasticity theory is then applied to phononic crystals involving a lattice of hollow cylinders, by making use of a dedicated perturbation approach. In particular, results highlight the influence of viscoelastic dissipation on the location of the first band gap. We show that dissipation shifts the band gap frequencies, simultaneously increasing the band gap width. These results are relevant to practical applications of soft viscoelastic solids subject to static pre-stress.