Across boreal forests, trees are the main living biomass carbon (C) stock, but the understory vegetation can contribute significantly to the C cycling and net forest carbon dioxide (CO2) balance. The patchy understory vegetation, which consists of sunlit (i.e. lichen‐like) and shaded habitats (i.e. dwarf shrub‐like), is often altered by ungulate grazers. Grazers may influence understory CO2 exchange and, consequently, the forest CO2 balance. Grazing affects differently the biomass of slow‐growing lichens compared to the faster‐growing mosses and dwarf shrubs, and therefore the effects of grazing on CO2 exchange in the patchy understory vegetation may vary temporally. We studied how excluding grazing for short and long periods affects the CO2 exchange and vegetation biomass in the understory of an oligotrophic Scots pine forest. We measured growing season (2019, 2020) CO2 exchange across sunlit and shaded habitats inside fences that had excluded large grazers for 0–1 and 25–26 years and in the adjacent grazed area. In addition, we measured the height of understory vegetation. We found that short‐term grazer exclusion increased ecosystem CO2 source fluxes only in the shaded habitats. However, long‐term exclusion of grazing decreased CO2 net release regardless of the habitat type. Furthermore, grazer exclusion increased moss depth immediately, which coincided with an abrupt intensification of CO2 net release. Considering the impacts of grazing over both short‐ and long‐term periods may help to forecast C fluxes more accurately, which may be relevant for informed climate solutions regionally and even on a larger scale.