Long-span bridges are susceptible to damage, aging, and deformation in harsh environments for a long time. Therefore, structural health monitoring (SHM) systems need to be used for reasonable monitoring and maintenance. Among various indicators, bridge displacement is a crucial parameter reflecting the bridge’s health condition. Due to the simultaneous bearing of multiple environmental loads on suspension bridges, determining the impact of different loads on displacement is beneficial for the better understanding of the health conditions of the bridges. Considering the fact that extreme gradient boosting (XGBoost) has higher prediction performance and robustness, the authors of this paper have developed a data-driven approach based on the XGBoost model to quantify the impact between different environmental loads and the displacement of a suspension bridge. Simultaneously, this study combined wavelet threshold (WT) denoising and the variational mode decomposition (VMD) method to conduct a modal decomposition of three-dimensional (3D) displacement, further investigating the interrelationships between different loads and bridge displacements. This model links wind speed, temperature, air pressure, and humidity with the 3D displacement response of the span using the bridge monitoring data provided by the GNSS and Earth Observation for Structural Health Monitoring (GeoSHM) system of the Forth Road Bridge (FRB) in the United Kingdom (UK), thus eliminating the temperature time-lag effect on displacement data. The effects of the different loads on the displacement are quantified individually with partial dependence plots (PDPs). Employing testing, it was found that the XGBoost model has a high predictive effect on the target variable of displacement. The analysis of quantification and correlation reveals that lateral displacement is primarily affected by same-direction wind, showing a clear positive correlation, and vertical displacement is mainly influenced by temperature and exhibits a negative correlation. Longitudinal displacement is jointly influenced by various environmental loads, showing a positive correlation with atmospheric pressure, temperature, and vertical wind and a negative correlation with longitudinal wind, lateral wind, and humidity. The results can guide bridge structural health monitoring in extreme weather to avoid accidents.