We estimate the weekly excess all-cause mortality in Norway and Sweden, the years of life lost (YLL) attributed to COVID-19 in Sweden, and the significance of mortality displacement. We computed the expected mortality by taking into account the declining trend and the seasonality in mortality in the two countries over the past 20 years. From the excess mortality in Sweden in 2019/20, we estimated the YLL attributed to COVID-19 using the life expectancy in different age groups. We adjusted this estimate for possible displacement using an auto-regressive model for the year-to-year variations in excess mortality. We found that excess all-cause mortality over the epidemic year, July 2019 to July 2020, was 517 (95%CI = (12, 1074)) in Norway and 4329 [3331, 5325] in Sweden. There were 255 COVID-19 related deaths reported in Norway, and 5741 in Sweden, that year. During the epidemic period of 11 March–11 November, there were 6247 reported COVID-19 deaths and 5517 (4701, 6330) excess deaths in Sweden. We estimated that the number of YLL attributed to COVID-19 in Sweden was 45,850 [13,915, 80,276] without adjusting for mortality displacement and 43,073 (12,160, 85,451) after adjusting for the displacement accounted for by the auto-regressive model. In conclusion, we find good agreement between officially recorded COVID-19 related deaths and all-cause excess deaths in both countries during the first epidemic wave and no significant mortality displacement that can explain those deaths.