Aims. This work aims to characterize the broadband emission of the nearby TeV blazar Mrk 501 during long periods of very low activity, and interpret such emission within widely used theoretical scenarios. Methods. We employ 4 years of multi-wavelength (MWL) observations of Mrk 501 performed from 2017 to 2020 with a multitude of instruments, involving, among others, MAGIC, Fermi-LAT, NuSTAR, Swift, GASP-WEBT, and OVRO. During this period, Mrk 501 showed an extremely low broadband activity, which may help to unravel its baseline emission. We characterize the dynamics of the emission through its multi-band fractional variability and correlations, and investigate the underlying high-energy particle population and emission mechanisms within a range of theoretical scenarios considering both leptonic and hadronic origins. Results. Despite the low broadband activity, significant flux variations are detected at all wavebands, with a trend of increasing variability with energy, and the highest flux variations occurring at X-rays and VHE γ-rays. A correlation between the X-rays and VHE γ-rays is measured with a statistical significance above 3σ, hence supporting leptonic scenarios to explain the most variable parts of the spectral energy distribution (SED), also during low activity states. Extending our data set to 12-years (from 2008 to 2020), we find significant correlations (> 3σ) between the X-ray and HE γ-ray emission on timescales that range from weeks to years, indicating, for the first time, a common physical origin driving the flux variability between these two bands. We additionally find a positive correlation between the HE γ-ray and radio fluxes, with the radio emission lagging the HE γ-ray emission by more than 100 days. This is consistent with the HE γ-ray emission zone being located upstream of the radiobright regions of the jet of Mrk 501. Furthermore, inside our 4-year data set, Mrk 501 showed a historically low activity in both X-rays and VHE γ-rays from mid-2017 to mid-2019 with a stable VHE flux (> 0.2 TeV) of ∼5% the emission of the Crab Nebula. The broadband SED of this 2-year long low-state period, the potential baseline emission of Mrk 501, can be adequately characterized with a one-zone leptonic model, a one-zone lepto-hadronic and a purely hadronic model that fulfill the neutrino flux constraints from IceCube. Furthermore, we explore the time evolution of the SED towards the historically low-state, revealing that the stable baseline emission may be ascribed to a standing shock, while the variable emission may be attributed to an additional expanding or traveling shock.