Recent observations have uncovered a correlation between the accretion rates (measured from the UV continuum excess) of protoplanetary discs and their masses inferred from observations of the sub-mm continuum. While viscous evolution models predict such a correlation, the predicted values are in tension with data obtained from the Lupus and Upper Scorpius star forming regions; for example, they underpredict the scatter in accretion rates, particularly in older regions. Here we argue that since the sub-mm observations trace the discs dust, by explicitly modelling the dust grain growth, evolution, and emission, we can better understand the correlation. We show that for turbulent viscosities with α ≲ 10−3, the depletion of dust from the disc due to radial drift means we can reproduce the range of masses and accretion rates seen in the Lupus and Upper Sco datasets. One consequence of this model is that the upper locus of accretion rates at a given dust mass does not evolve with the age of the region. Moreover, we find that internal photoevaporation is necessary to produce the lowest accretion rates observed. In order to replicate the correct dust masses at the time of disc dispersal, we favour relatively low photoevaporation rates ≲ 10−9 M⊙ yr−1 for most sources but cannot discriminate between EUV or X-ray driven winds. A limited number of sources, particularly in Lupus, are shown to have higher masses than predicted by our models which may be evidence for variations in the properties of the dust or dust trapping induced in substructures.