Modal tracking plays a vital role in structural health monitoring since changes in modal parameters help us understand a structure’s dynamic characteristics and may reflect the potential deterioration of structural performance. Although numerous modal parameter estimation (MPE) methods exist, it is not guaranteed that an MPE process will exclude all spurious modes and not lose any physical modes every time over a long-term monitoring period. Relatively large damping of a structure, poor data quality, and significant changes in structural modal parameters may make the estimated modal parameters spurious, missing, or misclassified. It makes long-term modal tracking semiautomated or manual, which constrains timely downstream applications such as anomaly detection, condition assessment, and decision making. This research aims to propose a long-term continuous automatic modal tracking algorithm based on Bayesian inference even when the modal parameters, damping, and data quality change significantly. Bayesian inference is used to determine the physical modes from the results of existing MPE methods. Both the modes identified from the most recent response set and the modal probability model from multiple previous response sets are considered in the Bayesian model to better determine the physical modes from the results of MPE. Moreover, the proposed algorithm requires only three extra hyperparameters compared to general modal tracking algorithms, and they can be quickly determined by a grid search method. The performance of the proposed algorithm is verified by a numerical example and a real-world civil structure Z24 Bridge benchmark.