Under carbon peaking and neutrality constraints, low-carbon retrofitting of residential buildings (LRRB) has become a strategic need for most countries worldwide. However, the development of China’s LRRB market still relies on government guidance without moving towards the goal of autonomous orientation. This area is still a concern for academics. Moreover, many stakeholders are involved in the LRRB, and the secondary stakeholders’ behavioral strategies do not substantially impact the LRRB. So, this paper adopts Mitchell’s score-based approach to identify the core stakeholders, followed by a tripartite evolutionary game model of government, ESCOs, and owners. Based on the system dynamics (SD) model, the evolution rules of the three parties’ behavior strategies and evolution stabilization strategies are analyzed, and the key factors influencing the equilibrium are found. The results of the study show that under the condition that the government adopts the same level of subsidy for ESCOs and owners, ESCOs are more sensitive to the subsidy; with the introduction of penalties under the premise of subsidy, ESCOs can reach evolutionary equilibrium faster; and when the benefits of owners accepting LRRB outweigh the losses, owners will eventually choose to accept retrofit regardless of whether the government subsidizes owners or not. Finally, the paper ends with suggestions for developing an LRRB market. The game model proposed in this paper can provide a scientific reference for stakeholders’ carbon reduction decisions.