Roads play a crucial role in urban transportation by facilitating the movement of materials within a city. The condition of road surfaces, such as damage and road facilities, directly affects traffic flow and influences decisions related to urban transportation maintenance and planning. To gather this information, we propose the Detecting and Clustering Framework for sensing road surface conditions based on crowd-sourced trajectories, utilizing various sensors (GPS, orientation sensors, and accelerometers) found in smartphones. Initially, smartphones are placed randomly during users’ travels on the road to record the road surface conditions. Then, spatial transformations are applied to the accelerometer data based on attitude readings, and heading angles are computed to store movement information. Next, the feature encoding process operates on spatially adjusted accelerations using the wavelet scattering transformation. The resulting encoding results are then input into the designed LSTM neural network to extract bump features of the road surface (BFRSs). Finally, the BFRSs are represented and integrated using the proposed two-stage clustering method, considering distances and directions. Additionally, this procedure is also applied to crowd-sourced trajectories, and the road surface condition is computed and visualized on a map. Moreover, this method can provide valuable insights for urban road maintenance and planning, with significant practical applications.