A defective sidewalk inhibits the walkability of a street and may also cause safety accidents (slips, trips, and falls) for pedestrians. When a pedestrian walks along a sidewalk, his/her behaviors may vary according to the condition of the sidewalk-e.g., whether the surface is normal, holed, cracked, tilted, or sloped. As a result, the pedestrian's stability may also change according to the built environment's conditions. Accordingly, this paper examines the feasibility of using pedestrians' physical behaviors to detect defects in a sidewalk. Pedestrians' physical responses and paths over a sidewalk are collected using an inertial measurement unit (IMU) sensor and a global positioning system (GPS). Then, after aggregating the pedestrians' bodily responses and locations, the irregularity of multiple pedestrians' responses are calculated in relation to their locations. The locations that show irregularities in the pedestrian-response patterns present a high correlation with the existence of a defect. The results of this study will help improve the continuous diagnosis of defects in sidewalks, thereby enhancing these built environment systems' serviceability.