Various concrete bridges have been built across oceans, valleys, and mountains; however, the settlement displacement of bridge piers caused by environmental changes or self-weight during construction phases often leads to uneven stresses, cracking, and eventual collapse. To address the labor-intensive and high-cost issues of pier displacement monitoring using contact-type sensors, this paper proposes an automatic vision-based method for measuring pier settlement displacement under complex construction environments, such as complex image backgrounds, varying ambient light, and camera movement. In the proposed method, a deep learning network was first employed to eliminate the adverse effect of complex image backgrounds and varying ambient light on the accuracy of target detection; then, an adaptive displacement extraction algorithm without a human-computer interaction process was developed to automatically extract the center coordinates of targets attaching to the bridge piers and reference platform; finally, the pier settlement displacement was calculated by using the relative displacements obtained by a dual camera system to eliminate the measurement error caused by camera translation and rotation movements. Laboratory tests of a cantilever beam and field tests of a continuous multispan concrete girder highway bridge under construction have successfully validated the effectiveness and robustness of the developed methodology. The results obtained in this paper can provide some insights for engineers in applying computer vision technology for the real-time monitoring of bridge displacements.