Preliminary analysis of satellite measurements from around the world showed drops in nitrogen dioxide (NO<sub>2</sub>) with lockdowns due to the COVID-19 pandemic. A number of studies have found these drops to be correlated with local decreases in transportation and/or industry. None of these studies, however, has rigorously quantified the statistical significance of these drops relative to natural meteorological variability and other factors that influence pollutant levels during similar time periods in previous years. Here, we develop a novel statistical testing framework that accounts for seasonal variability, transboundary influences, and new factors such as COVID-19 restrictions in explaining trends in several pollutant levels at 16 ground-based measurement sites in Southern Ontario, Canada. We find statistically significant and temporary drops in NO<sub>2</sub> (11 out 16 sites) and CO (all 4 sites) in April-June 2020, with pollutant levels 20% lower than in the previous three years. Much fewer sites (2-3 out of 16) experienced statistically significant drops in O<sub>3</sub> and PM2.5.<b> </b>The statistical testing framework developed here is the first of its kind applied to air quality data, and highlights the need for rigorous assessment of statistical significance, should analyses of pollutant level changes post COVID-19 lockdowns be used to inform policy decisions in Ontario, Canada. See Methods section in the manuscript.