Since the discovery of surface-enhanced Raman scattering (SERS) 40 years ago, the origin of the "background" that is systematically observed in SERS spectra has remained questionable. To deeply analyze this phenomenon, plasmon-resonant Raman scattering was recorded under specific experimental conditions on a panel of composite multilayer samples containing noble metal (Ag and Au) nanoparticles. Stokes, anti-Stokes, and wide, including very low, frequency ranges have been explored. The effects of temperature, size (in the nm range), embedding medium (SiO 2 , Si 3 N 4 , or TiO 2 ) or ligands have been successively analyzed. Both lattice (Lamb modes and bulk phonons) and electron (plasmon mode and electron-hole excitations) dynamics have been investigated. This work confirms that in Ag-based nanoplasmonics composite layers, only Raman scattering by single-particle electronic excitations accounts for the background. This latter appears as an intrinsic phenomenon independently of the presence of molecules on the metallic surface. Its spectral shape is well described by revisiting a model developed in the 1990s for analyzing electron scattering in dirty metals, and used later in superconductors. The g s factor, that determines the effective mean-free path of free carriers, is evaluated, g expt s = 0.33 ± 0.04, in good agreement with a recent evaluation based on time-dependent local density approximation g theor s = 0.32. Confinement and interface roughness effects at the nanometer range thus appear crucial to understand and control SERS enhancement and more generally plasmon-enhanced processes on metallic surfaces.