Coal-fired heating is the main method of heating in China, causing serious air pollution and large amounts of CO2 emissions. Decarbonizing heating is important to reduce carbon emissions, and choosing a suitable heating technical scheme is conducive to the early realization of carbon neutrality in China. Coal to gas and coal to electricity transformation projects were carried out in 2017 and achieved remarkable effects. This study compares the current domestic and international clean heating modes, where gas heating, electric heating, heat hump heating, biomass heating, and solar heating coupling system are taken into account. The heating technology potential and heating support aspects, including the industrial sector, building sector, carbon capture and storage (CCS) technology, and publicity are explored as well. Regarding the actual situation in China, a comparative analysis is also conducted on the different types of heat pumps, and then an optimal heating scheme for urban and rural areas is proposed. It is suggested that the urban area with centralized heating can install ground source heat pumps, and the rural area with distributed heating can apply a coupling system of solar photovoltaics to ground source heat pumps (PV-GSHP). Based on current policies and standards support, this study calculates the carbon emissions of this scheme in 2030 and provides a detailed analysis of relevant parameters. The feasibility and superiority of the scheme are confirmed by comparison and discussion with other studies. Moreover, specific measures in planning, subsidy, construction, and electricity are proposed to implement the heating scheme. This study provides a reference for the mode selection and technical scheme of heating decarbonation in China, and that could be also considered in other regions or countries.