Psychological hazards within organizational structures of construction sites are difficult to detect and can have significant negative impacts on safety performances when such hazards erupt. At present, most safety performance assessment models for construction sites ignore psychological factors. Therefore, in order to reveal psychological hazards within construction site organizations and to avoid damage caused by psychological hazards to safety performances, this paper evaluates the safety performances of construction sites by focusing on leader–member exchange ambivalence as the main trigger point. The evaluation system and evaluation criteria are established through three aspects: building scale, emotional orientation, and stability factors. The hierarchical analysis method, game theory, and extension cloud model are combined to make evaluation results more objective and credible. Moreover, a construction project with high technical requirements, high investment, and complex construction conditions (defined as a complex project) and an ordinary construction project with low technical difficulty and simple construction conditions (defined as a general project) were selected for analysis. The evaluation results indicate that both complex projects and general projects have safety hazards regarding psychological orientations. Finally, this paper makes some suggestions from three aspects: management system and corporate culture, building site intelligence, and social opinion to improve the safety performances of construction sites. The evaluation results are the same as actual operation results, which verify that models proposed in this paper can be used for safety performance evaluations of actual construction projects and provide help for managers to grasp overall safety levels.