It has been argued that recycled gas from stellar mass loss in galaxies might serve as an important fuelling source for black holes (BHs) in their centers. Utilizing spectroscopic samples of galaxies from the Sloan Digital Sky Survey (SDSS) at z = 0–0.35 and the Large Early Galaxy Astrophysics Census (LEGA-C) survey at z = 0.6–1 that have X-ray coverage from XMM-Newton or Chandra, we test this stellar mass loss fuelling scenario by investigating how AGN activity and BH growth vary with the break strength at 4000 Å, $\rm D_{n}4000$ (which is closely related to the age of stellar populations), as younger galaxies are considered to have higher stellar mass loss rates. We found that when controlling for host-galaxy properties, the fraction of log LX/M⋆ >32 (which roughly corresponds to Eddington ratios ≳ 1%) AGN and sample-averaged black hole accretion rate ($\rm \overline{BHAR}$) decrease with $\rm D_{n}4000$ among $\rm D_{n}4000$ ≲ 1.9 galaxies, suggesting a higher level of AGN activity among younger galaxies, which supports the stellar mass loss fuelling scenario. For the oldest and most massive galaxies at z = 0–0.35, this decreasing trend is not present anymore. We found that, among these most massive galaxies at low redshift, the fraction of low specific-accretion-rate (31 < log LX/M⋆ < 32) AGNs increases with $\rm D_{n}4000$, which may be associated with additional fuelling from hot halo gas and/or enhanced accretion capability.