This study addresses the stress–strain relaxation functions of solid polymers in the framework of the linear viscoelasticity with aim to establish the adequate fractional operators emerging from the hereditary integrals. The analysis encompasses power-law and non-power-law materials, thus allowing to see the origins of application of the tools of the classical fractional calculus with singular memory kernels and the ideas leading towards fractional operators with non-singular (regular) kernels. A step ahead in modelling with hereditary integrals is the decomposition of non-power-law relaxation curves by Prony series, thus obtaining discrete relaxation kernels with a finite number of terms. This approach allows for seeing the physical background of the newly defined Caputo–Fabrizio time fractional derivative and demonstrates how other constitutive equations could be modified with non-singular fading memories. The non-power-law relaxation curves also allow for approximations by the Mittag–Leffler function of one parameter that leads reasonably into stress–strain hereditary integrals in terms of Atangana–Baleanu fractional derivative of Caputo sense. The main outcomes of the analysis done are the demonstrated distinguishes between the relaxation curve behaviours of different materials and are therefore the adequate modelling with suitable fractional operators.