ImportanceHong Kong was held as an exemplar for pandemic response until it recorded the world’s highest daily COVID-19 mortality, which was likely due to vaccine refusal. To prevent this high mortality in future pandemics, information on underlying reasons for vaccine refusal is necessary.ObjectivesTo track the evolution of COVID-19 vaccination willingness and uptake from before vaccine rollout to mass vaccination, to examine factors associated with COVID-19 vaccine refusal and compare with data from Singapore, and to assess the population attributable fraction for vaccine refusal.Design, Setting, and ParticipantsThis cohort study used data from randomly sampled participants from 14 waves of population-based studies in Hong Kong (February 2020 to May 2022) and 2 waves of population-based studies in Singapore (May 2020 to June 2021 and October 2021 to January 2022), and a population-wide registry of COVID-19 vaccination appointments. Data were analyzed from February 23, 2021, to May 30, 2022.ExposuresTrust in COVID-19 vaccine information sources (ie, health authorities, physicians, traditional media, and social media); COVID-19 vaccine confidence on effectiveness, safety, and importance; COVID-19 vaccine misconceptions on safety and high-risk groups; political views; and COVID-19 policies (ie, workplace vaccine mandates and vaccine pass).Main Outcomes and MeasuresPrimary outcomes were the weighted prevalence of COVID-19 vaccination willingness over the pandemic, adjusted incidence rate ratios, and population attributable fractions of COVID-19 vaccine refusal. A secondary outcome was change in daily COVID-19 vaccination appointments.ResultsThe study included 28 007 interviews from 20 waves of longitudinal data, with 1114 participants in the most recent wave (median [range] age, 54.2 years [20-92] years; 571 [51.3%] female). Four factors—mistrust in health authorities, low vaccine confidence, vaccine misconceptions, and political views—could jointly account for 82.2% (95% CI, 62.3%-100.0%) of vaccine refusal in adults aged 18 to 59 years and 69.3% (95% CI, 47.2%-91.4%) of vaccine refusal in adults aged 60 years and older. Workplace vaccine mandates were associated with 62.2% (95% CI, 9.9%-139.2%) increases in daily COVID-19 vaccination appointments, and the Hong Kong vaccine pass was associated with 124.8% (95% CI, 65.9%-204.6%) increases in daily COVID-19 vaccination appointments.Conclusions and RelevanceThese findings suggest that trust in health authorities was fundamental to overcoming vaccine hesitancy. As such, engendering trust in health care professionals, experts, and public health agencies should be incorporated into pandemic preparedness and response.