The rapid development of the metro has greatly relieved the traffic pressure on the urban ground system, but the frequency of metro construction accidents is also increasing year by year. Due to the complex construction process of the metro, once an accident occurs, casualties and property damage are extremely serious. The safety risk factors triggered by different stakeholders were the primary cause of accidents during the metro construction phase. This paper builts a social analysis network of safety risk factors in metro construction from a stakeholder’s perspective. Based on 42 accident cases and related literature, 6 stakeholders and 25 safety risk factors were identified and the relationships between stakeholders and safety risk factors were also determined. Through the application of social network analysis, a social network of safety risk factors in metro construction was constructed, and quantitative analysis was carried out based on density, degree centrality, betweenness centrality, and cohesive subgroup. The results showed that the key safety risk factors in the construction phase of the metro were in action of the contractor’s construction site managers, lack of safety protection at the construction site, insufficient detailed survey and design information provided by the designer, unfavorable government regulation, and bad weather. Moreover, the results of 20 cohesive subgroups illustrated the interrelationship between safety risk factors. S1H2 (“violations by operatives” related to contractor) and S1H4 (“lack of safety precautions” related to contractor) and S5H5 (“ineffective supervision” related to supervisor) both belonged to subgroup G1, which means that there is a high probability that these three safety risk factors would occur simultaneously. This paper provided a basis to improve the level of safety risk management and control from the stakeholder’s perspective.