To identify the influencing factors and internal mechanism of knowledge transfer among prefabricated construction workers (PCWs), this study constructs a knowledge transfer behavior model for PCWs from various perspectives, including behavioral attitude, subjective norms, physiological perceived control, behavioral habits, and behavioral intention based on a modified Theory of Planned Behavior (TPB). It also employs a Structural Equation Model (SEM) for hypothesis validation and utilizes System Dynamics (SD) to simulate the knowledge transfer process of PCWs. Through empirical and simulation research, three conclusions are drawn: (1) Knowledge transfer willingness has a promoting effect on knowledge transfer behaviors. (2) Trusted relationships, organizational culture, physiological perceived control, and behavioral habits all have a promoting effect on workers’ knowledge transfer willingness and subsequently promote their knowledge transfer behaviors. (3) Among these factors, the strength of knowledge transfer willingness has the most significant impact on workers’ behavioral habits, followed by trusted relationships, physiological perceived control, and organizational culture. Additionally, when comparing the results of this study with knowledge transfer influencing factors of traditional construction workers (TCWs), it is found that trusted relationships and self-efficacy have a promoting effect on the knowledge transfer willingness of PCWs as well as TCWs. However, the impact of organizational culture, environmental perception, and behavioral habits on the knowledge transfer willingness of PCWs is more pronounced compared to TCWs. Based on TPB, this study constructs a suitable theoretical model to analyze the knowledge transfer process of PCWs by mining their group characteristics, and the research results establish a systematic analysis framework for the study of knowledge transfer behavior of PCWs. It also provides scientific suggestions for the formulation of targeted measures to enhance the willingness and efficiency of knowledge transfer of PCWs.