Damage detection of a complex spatial frame structure with semi-rigid joints is a challenging task. Most existing damage detection methods consider the joint as rigid and the nonuniform cross section of the member is not considered. This assumption results to a large error in the structural damage identification. A novel generic element is proposed for the nonuniform cross-section member with semi-rigid joints at both ends in this study. The finite element model of spatial structures with semi-rigid joints is established using the proposed element. The modal strain energy-based damage index is defined and its sensitivity to the member and joint damage has been investigated using numerical and experimental study. An 8-m long bridge model with the bolted connection at joints is built in the laboratory. Dynamic responses of the bridge under random excitations are monitored using 13 wireless tri-axis accelerometers. The spatial mode shapes of the bridge are extracted using the reference-based stochastic subspace identification from multiple set wireless measurements. The numerical model is validated using experimental results. Dynamic responses of the bridge with different damage scenarios have been simulated using the validated model and the corresponding damage indices are obtained. The results show that the proposed method is reliable and accurate to analyze dynamic behavior of the spatial structure with semi-rigid joints, and structural damage can be identified accurately using the modal strain energy-based damage index.