We estimated the impact of the COVID-19 pandemic on mortality in Brazil for 2020 and 2021 years. We used mortality data (2015-2021) from the Health Ministry, Brazil government, to fit linear mixed models for forecasting baseline deaths under non-pandemic conditions. An advantage of the linear mixed model is the flexibility to capture year-trend while dealing with the correlations among death counts over time. Following a specified model building strategy, estimation of all-cause excess deaths at the country level and stratified by sex, age, ethnicity and region of residence, from March 2020 to August 2021. We also considered the estimation of excess deaths by specific causes. Estimated all-cause excess deaths was 199,108 (95% PI: 171,007; 227,209, P-Score=17.3%) for weeks 10-53, 2020, and 417,167 (95% PI: 372,075; 462,259, P-Score=50.1%) for weeks 1-32, 2021. P-scores ranged from 5.4% (RS, South) to 36.2% (AM, North) in 2020 and from 29.3% (AL, Northeast) to 94.9%$ (RO, North) in 2021. Differences among men (18.9%) and women (14.2%) appeared in 2020 only, and the P-scores were about 51% for both sexes in 2021. Except for youngsters (<20 years old), all adult age groups were badly hit, especially those from 40 to 79 years old. In 2020, the Indigenous+East Asian population had the highest P-score (27%), and the Black population suffered the greatest impact (61.9%) in 2021. The pandemic impact had enormous regional heterogeneity and substantial differences according to socio-demographic factors, mainly during the first wave, showing some population strata benefits from the social distancing measures when able to adhere to them. In the second wave, the burden was very high for all but extremely high for some, highlighting our society needs to tackle the health inequalities experienced by groups of different socio-demographic and economic status.