Buildings are among the largest sources of carbon emissions, so their owners have been compelled to set stringent sustainability goals in response to the global call for carbon neutrality. Consequently, facility managers are pressed to make their buildings smarter through adopting greater value-for-money sustainability solutions. Besides the implementation of traditional energy-saving measures, retrofitting buildings with smart building (SB) features, known as ‘smart retrofitting’ (SR), has become a growing trend. However, there is still no known set of appropriate criteria for making decisions on SR. To fill this void, a study was undertaken, where a four-tier hierarchical framework of SR decision-making criteria has been established after a thorough literature search on the Web of Science database followed by an in-depth qualitative review and a focus group discussion. A total of 32 criteria were identified, which fall into five categories: (1) Financial; (2) Technical; (3) Human; (4) Environmental; and (5) Legal, safety, and security. A key finding is that it is not merely the post-retrofit performance but also the retrofitting process itself that plays an important part in SR decision-making. The identified criteria demonstrate that ‘smartness’ encompasses not only ‘energy efficiency’ but also a vast array of other technical and social factors. The broad scope portrayed by the criteria, which needs to be considered holistically, provides guidance to facility managers in making SR decisions. Future works for refining the SR decision-making framework are also outlined.