Widefield quantum microscopy based on nitrogen-vacancy (NV) centres in diamond has emerged as a powerful technique for quantitative mapping of magnetic fields with a sub-micron resolution. However, the accuracy of the technique has not been characterised in detail so far. Here we show that optical aberrations in the imaging system may cause large systematic errors in the measured quantity beyond trivial blurring. We introduce a simple theoretical framework to model these effects, which extends the concept of a point spread function to the domain of spectral imaging. Using this model, the magnetic field imaging of test magnetic samples is simulated under various scenarios, and the resulting errors quantified. We then apply the model to previously published data, show that apparent magnetic anomalies can be explained by the presence of optical aberrations, and demonstrate a post-processing technique to retrieve the source quantity with improved accuracy. This work presents a guide to predict and mitigate aberration induced artefacts in quantitative NV-based widefield imaging and in spectral imaging more generally.