Although many studies have been carried out on pedestrian crossing safety, comprehensive research evaluating vehicle–pedestrian conflict in areas with zebra crossing (AWZCs) versus areas without zebra crossing (AWOZCs) is still neglected. In the present study, through a naturalistic driving study (NDS), drivers’ behavior was recorded in AWZCs and AWOZCs. Vehicle–pedestrian conflicts were evaluated by examining the evasive maneuver behavior of drivers and pedestrians based on surrogate measures of safety (SMoS). The severity of conflicts was categorized by a K-means clustering method into three specific groups based on the critical thresholds of SMoS. The evasive maneuvers performed by pedestrians and drivers were classified into three levels: normal, slight, and serious. In conflicts resulting in normal and serious maneuvers, drivers would attempt to prevent collisions by changing the speed and direction of the vehicle. Moreover, a pedestrian at the slight level of conflict was the determinative factor in reducing the possibility of collisions by performing actions such as returning to the curb of the street or increasing walking speed. Also, the results showed that pedestrians were more likely to cross with a smaller margin of safety in AWOZCs than in AWZCs. This study explains how both pedestrians and drivers play a crucial role in preventing collisions during different levels of conflict. Given this finding, conducting future research to evaluate the interaction between drivers and pedestrians may lead to establishing a basic framework for designing an algorithm to detect the possibility of a pedestrian collision.