Bridge health monitoring (BHM) is important due to its benefits in detecting and diagnosing potential damages to bridges, providing early warning signals, and guiding maintenance decisions. Compared to manual inspection and fixed-sensors-based monitoring, drive-by BHM leverages vibration responses measured from vehicles passing over bridges to indirectly diagnose bridge damages, offering a rapid, mobile, and economical complementary solution. However, vehicle-bridge interaction (VBI) systems have large variations in bridge configurations, vehicle suspension systems, driving speeds, and so on, making it challenging to develop a damage diagnosis algorithm that is robust to vehicle-bridge variability. Moreover, existing approaches often require vehicle speed within a specific range to provide both informative and reliable signals, limiting their practical applications. To address these challenges, we introduce a damage diagnosis approach that extracts damage-sensitive features and enhances their robustness to vehicle-bridge variability through physics-informed signal decomposition. Our approach first pre-processes and decomposes the vehicle vibration signal using the synchro-squeezed wavelet transform (SWT) because of its anti-noise property and its ability to represent the non-stationary and time-varying signals from drive-by vehicles. Then, a damage-sensitive signal is reconstructed using the inverse SWT within a physics-informed frequency band which excludes the vehicle and bridge resonances while keeping the damage-sensitive information. Peak features, such as peak location and energy, are input to the Gaussian Mixture Model clustering algorithm for diagnosing bridge damage in an unsupervised fashion. The performance of the proposed approach is evaluated on a numerical VBI model, which includes three vehicle types, six bridge lengths, and three bridge cross-sections, and takes into consideration the variability in vehicle properties and speed. The results validate that the extracted features are damage-sensitive and robust to various vehiclebridge systems, achieving an overall mean absolute percentage error of 0.78% for damage localization and 13.13% for damage quantification.