Urban policies have recently been formulated, following the increasing interest in pedestrian-friendly cities, people-centered safety, and accessibility. Despite the research efforts on physical walking safety, safety evaluations centered on pedestrian perception have been under-reported. Investigating the factors affecting pedestrian subjective safety perception is critical to promoting walking intention because pedestrians forgo walking if they feel unsafe. This study explored the relationship between various walking environmental factors and pedestrians’ psychological perception of safety by surveying 99 pedestrians’ perceptions at nine study sites and conducting a field investigation. Because of the multifaceted nature of pedestrian perception, mediation effect analyses were also conducted to understand the relationship between walking environment factors and perceived safety in depth, considering the role of the perception of traffic characteristics and walking infrastructure. This study found that walking environmental factors closely related to physical safety (e.g., traffic safety facilities and crosswalks) may not greatly contribute to perceived safety and demonstrated that maintaining infrastructure quality is essential for enhancing perceived safety, considering the mediating effect of the perception of infrastructure on perceived safety. The results imply that to improve the walking environment, it is necessary to consider both the physical safety and the perceived safety of pedestrians. This requires comprehensive planning for enhancing traffic safety facilities as well as ensuring user comfort and pleasure through quality infrastructure. This study can provide a basis for enhancing pedestrian-centered safety and promoting residents’ walking intention for public health while increasing their perceptions of safety.