Considering a large number of Small and Medium-Scale Contractors (SMSCs) dominating the construction industries of many developing nations and the increasingly high rate of tower crane accidents in the industry, this paper seeks to methodically determine the safety risk factors that are significant in influencing construction site safety, especially on sites where the SMSCs operate tower cranes. The paper will further assess the extent to which each safety risk factor affects safety on construction sites. Data for the study was obtained through a literature search, site visual observations, discussions with site operators, and structured questioning of safety and equipment managers of leading construction companies. The study's findings reveal that the operator's low experience level was the most significant factor influencing construction site safety, particularly when operating tower cranes. This paper systematically investigates the key safety risk factors that influence construction site safety when operating tower cranes. The paper presents a clear methodology for identifying and prioritizing the safety risk factors, which may readily apply to other construction equipment. The findings of this paper are expected to play an important role in promoting and enhancing a safety culture for operating tower cranes in construction sites, particularly the project sites operated by the SMSCs.