Constructional material deterioration and member damage can cause changes in the dynamic characteristics of bridge structures, and such changes can be tracked in the responses of passing vehicles via the vehicle-bridge interaction (VBI). Though data-driven methods have shown promising results in damage inspection for drive-by methods, there is still much room for improvement in their performance. Given this background, this paper proposes a novel time-domain signal processing algorithm for the raw vehicle acceleration data of data-driven drive-by inspection methods. To achieve the best data processing performance, an optimizing strategy is designed to automatically search for the optimal parameters, tuning the algorithm. The proposed method intentionally overcomes the difficulties in the application of drive-by methods, such as measurement noise, speed variance, and enormous data volumes. Meanwhile, the use of this method can greatly improve the accuracy and efficiency of Machine Learning (ML) models in vehicle-based damage detection. It consists of a filtering process to denoise the data, a pooling process to reduce data redundancy, and an optimizing procedure to maximize algorithm performance. A dataset is obtained to validate the proposed algorithm through laboratory experiments with a scale truck model and a steel beam. The results show that, compared to using raw data, the present algorithm can increase the average accuracy by 12.2–15.0%, and the average efficiency by 35.7–96.7% for different damaged cases and ML models. Additionally, the functions of filtering and pooling operations, the influence of window function parameters, as well as the performance of different sensor locations, are also investigated in the paper. The goal is to present a signal processing algorithm for data-driven drive-by inspection methods to improve their detection performance of bridge damage caused by material deterioration or structural change.