COVID-19, which was named in March 2020 as a global pandemic by the WHO, remains a serious public health threat worldwide. Despite the adoption of vaccines as an effective strategy to counter this pandemic, the vaccination rate in Cameroon is far lower than that planned by the Cameroonian government and its partners. The main objective of this study was to determine the factors limiting COVID-19 vaccine acceptance in the Menoua Division in the West Region of Cameroon. A community-based cross-sectional and analytical study was conducted between March and April 2022 in the Menoua Division. A pre-tested questionnaire was filled out by willing participants of more than 18 years old, and data were further expressed in order to estimate the knowledge of participants on COVID-19, vaccine status, and the factors associated with vaccine refusal. A Pearson test was performed in order to identify the associated factors, with a p-value < 0.05 considered as significant. A total of 520 participants with a mean age of 33.27 ± 12.78 were included. Most had a secondary education level (56.15%), and trade and informal sectors (34.04%) were the main occupations. Knowledge on COVID-19 was average, and it was significantly associated (p < 0.05) with gender and education level. The vaccination rate was 10%, which was six times less than the national target. A lack of information, confidence, and medicinal plant use were all factors significantly associated with vaccine refusal. This pioneer community-based study in Cameroon identified a lack of knowledge, confidence, and medicinal plant use as the leading factors limiting COVID-19 vaccine acceptance in Cameroon. Health authorities should therefore strengthen sensitization in order to tackle the lack of information and the misinformation among the target groups.