In order to improve the detection accuracy of urban tram track obstacles running in complex environments, a track obstacle detection algorithm based on BF-SSD is proposed. Design convolutional splitting structure, replace all 3*3 convolutional layers in VGG16 except the first convolutional layer, improve the detection speed, design two-way fusion modules, strengthen the feature expression ability of the low-level feature layer, improve the detection accuracy of small targets, design a two-stage deconvolution module, make up for the noise information generated by the high-level feature layer due to excessive deconvolution, enhance the detection effect in harsh environments, propose to improve NMS, and improve the detection ability of coincident targets. Through the test and analysis of the obstacle image dataset of the self-made normal environment and the harsh environment, the experimental results show that compared with other algorithms, the loss function of the BF-SSD algorithm is the lowest, and the mAP of the BF-SSD algorithm reaches 90.03% and 87.83% respectively under normal weather and bad weather environments, and the mAP of the detection of small targets reaches 70.26% and 69.26%, respectively. The BF-SSD algorithm improves the detection accuracy of tram obstacles, especially the detection ability of small target obstacles.