Prefabricated buildings (PBs) are a new type of building construction, which are less time-consuming and cause low environmental pollution and resource consumption. They play an important role in industrialized construction and clean production and have gained worldwide attention. However, the high construction costs have become a major obstacle to their popularity and application. This study investigates the factors influencing construction costs of PBs in China using a systematic literature review (SLR), fuzzy interpretive structure modeling (fuzzy ISM), and the Matrice d’Impacts croises-multiplication appliqué an classment (MICMAC) technique. First, 32 influencing factors were identified from the SLR. Second, out of which 16 critical factors were selected and mapped in a hierarchical model through semi-structured interview screening, and the MICMAC technique was used to classify the cost-influencing factors of PBs into different categories. The results revealed that all identified factors played pivotal roles in various capacities and influenced the cost of PB construction. This study may assist administrators and policymakers in better understanding the factors that influence the costs of PBs construction to manage and reduce them.