We sought to analyze the relationship between health literacy, confidence in COVID-19 vaccines, and self-reported vaccination. We hypothesized that the relationship between health literacy and vaccination would be mediated by vaccine confidence. We recruited (N = 271) English- and Spanish-speaking adults in Boston and Chicago from September 2018 to September 2021. We performed a probit mediation analysis to determine if confidence in COVID-19 vaccines and health literacy predicted self-reported vaccination. We hypothesized that the relationship between health literacy and vaccination would be mediated by vaccine confidence. Participants were on average 50 years old, 65% female, 40% non-Hispanic Black, 25% Hispanic, and 30% non-Hispanic White; 231 (85%) reported at least one COVID-19 vaccination. A higher mean vaccine confidence score (t = −7.9, p < 0.001) and higher health literacy (t = −2.2, p = 0.03) were associated with vaccination, but only vaccine confidence predicted vaccination in a multivariate model. Vaccine confidence mediated the relationship between health literacy and COVID-19 vaccination (mediated effects: 0.04; 95% CI [0.02, 0.08]). We found that using a simple tool to measure vaccine confidence identified people who declined or delayed COVID-19 vaccination in a diverse sample of adults with varying levels of health literacy. Simple short survey tools can be useful to identify people who may benefit from vaccine promotion efforts and evidence-based communication strategies.