Bridge structures are susceptible to environmental and operational variations (EOVs). Improperly handling these influences may result in incorrect assessments of the bridge’s health condition. Blind source separation (BSS) techniques show promising potential in suppressing the effects of EOVs. However, major challenges such as high data variability, difficulty in parameter selection, lack of reliable decision thresholds, and practical engineering validation have seriously hindered the application of such techniques in bridge health monitoring. Consequently, this paper proposes a new method for bridge damage detection that combines complexity pursuit (CP) and extreme value theory (EVT). This method first uses the exponentially weighted moving average (EWMA) technique to preprocess the measured modal frequencies. The CP algorithm and information entropy are then used to extract structural damage sources from the preprocessed data automatically. Based on the extracted structural damage sources, the damage index (DI) is defined using k-means clustering and Euclidean distance. Following that, the generalized extreme value (GEV) distribution is used to fit the DI data under the normal condition of the bridge, and the damage detection threshold is given according to the fitted distribution. Benchmark data of the KW51 railway bridge are considered to verify the effectiveness of the proposed method along with several comparative studies. The results show that even under strong EOV influences, the proposed method still maintains good damage detection accuracy and robustness, and its effectiveness is superior to some well-known damage detection methods.