Utilizing heat pumps has varied benefits, including decreasing the proportion of fossil fuels in the energy mix and reducing CO2 emissions compared with other heating modes. However, this effect greatly depends on the type of external energy and the type of the applied heat pump system. In our study, two different types of heat pumps, three different modes of operation, three different types of auxiliary energy, and three different CO2 emission values from electricity generation were selected to calculate the CO2 emissions related to heating a theoretical house and calculate the CO2 emissions reduction compared with gas firing. According to the calculations, a wide range of CO2 emission reductions can be achieved, from scenarios where there is no reduction to scenarios where the reduction is 94.7% in monovalent mode. When operating in a bivalent mode, the values are less favorable, and several systems show no reduction, particularly when operating in an alternate mode at a bivalent temperature of 2 °C. However, the reduction in fossil CO2 emissions can be kept at a high value (up to 56.7% with Hungary’s electricity mix) in a bivalent system by using biomass as a resource of auxiliary energy and geothermal heat pumps, which is very similar to the CO2 emission reduction in monovalent systems (54.1%).