Studies have identified teacher effects on students’ attitudes. This study explored the differences in and associations between teachers’ and students’ COVID-19 vaccine hesitancy, and the factors that affect it. A population-based self-administered online survey was conducted to evaluate the COVID-19 vaccine hesitancy of teachers and students in a college in Taizhou, China. A total of 835 valid questionnaires were obtained. All data were analyzed using IBM SPSS statistics 26.0 software. The proportions of COVID-19 vaccine hesitancy for teachers and students were 31.7%, and 23.8%, respectively. In the binary logistic regression analysis, teachers who perceived the safety of the COVID-19 vaccine as low (OR = 6.794 , 95%CI: 1.701–27.143), did not pay continuous attention to the vaccine news (OR = 3.498, 95%CI: 1.150–10.640), and suffered chronic diseases (OR = 2.659, 95%CI: 1.135–6.227) were more likely to hesitate to get vaccinated against COVID-19. The group of students who perceived the COVID-19 vaccine safety as being low (OR = 1.805, 95%CI: 1.094–2.979) were more hesitant to get the COVID-19 vaccine. While both teachers and students were hesitant about the COVID-19 vaccine, teachers were found to be more so. Perceptions regarding the safety of COVID-19 vaccines, attention to and awareness of vaccine news, and chronic medical conditions were the main factors that influenced the hesitation regarding COVID-19 vaccines. Therefore, students’ vaccine hesitancy may depend largely on the perceptions of the vaccine’s safety rather than teachers’ vaccine hesitancy.