The condition and hazard monitoring of bridges play important roles in ensuring their service continuity not only throughout their entire lifespan but also under extreme conditions such as those of earthquakes. Advanced structural health monitoring (SHM) systems using vision-based technology, such as surveillance, traffic, or drone cameras, may assist in preventing future impacts due to structural deficiency and are critical to the emergence of sustainable and smart transportation infrastructure. This study evaluates several feature detection and tracking algorithms and implements them in the vision-based SHM of bridges along with their systematic procedures. The proposed procedures are implemented via a two-span accelerated bridge construction (ABC) system undergoing a large-scale shake-table test. The research objectives are to explore the effect of refined matching algorithms on blob-based features in improving their accuracies and to implement the proposed algorithms on large-scale bridges tested under seismic loads using vision-based SHM. The procedure begins by adopting blob-based feature detectors, i.e., the scale-invariant feature transform (SIFT), speeded-up robust features (SURF), and KAZE algorithms, and their stability is compared. The least medium square (LMEDS), least trimmed square (LTS), random sample consensus (RANSAC), and its generalization maximum sample consensus (MSAC) algorithms are applied for model fitting, and their sensitivity for removing outliers is analyzed. The raw data are corrected using mathematical models and scaled to generate displacement data. Finally, seismic vibrations of the bridge are generated, and the seismic responses are compared. The data are validated using target-tracking methods and mechanical sensors, i.e., string potentiometers. The results show a good agreement between the proposed blob feature detection and matching algorithms and target-tracking data and reference data obtained using mechanical sensors.