Computer vision is becoming one of the most popular remote-sensing techniques and has been used widely in displacement monitoring and damage identification of in-service bridges. Nevertheless, several obstacles, including limited sampling rate, insufficient resolution for remote measurement, and error induced by camera tilting, restrict the application of vision-based approaches in structural health monitoring (SHM). The combination of a traditional SHM system and a modern remote-sensing technique can significantly improve the accuracy and reliability of the monitoring system. To make full use of data collected in the traditional SHM system and computer vision technique and overcome their shortcomings, we presented an improved bridge displacement estimation approach for SHM purposes by fusing camera-based and acceleration measurements. First, we estimated the scaling factor, which transfers pixel displacement to real displacement under tilt photogrammetry, by the acceleration reconstructed and camera-based displacements in the same frequency band without the actual size of the structure or the measurement parameters. Then, we extracted the low-frequency displacement from the vision-based measurement, and we fused the high-frequency displacement that was reconstructed from the acceleration measurement to achieve high-accuracy displacement estimation. The efficiency of this method was validated through dynamic load tests on a suspension model bridge in the laboratory and field tests on a highway and subway cable-stayed bridge.