Background The Coronavirus Infectious Disease 2019 (COVID-19) pandemic has continuously affected human life with several devastating effects. Currently, there are effective vaccines to protect people from COVID‐19 and the World Health Organization (WHO) has highlighted strategies to influence COVID-19 vaccine uptake in hard-to-reach communities in Ghana. However, prior studies on COVID-19 vaccine acceptability in Ghana are online surveys targeting the literates and those in urban areas, leaving residents in far-flung communities. We assessed knowledge, attitude and acceptability of COVID-19 vaccine among residents in rural communities in Ghana. Methods This study was a community-based cross-sectional study and was conducted at three selected regions in Ghana (Northern, Ashanti and Western North) from May to November, 2021. This study included residents 15–81 years, living in the selected rural communities for more than 1 year. Study participants were recruited and questionnaires administered to collect data on knowledge, attitude and acceptance of the COVID-19 vaccine. Statistical analyses were performed using Statistical Package for Social Science (SPSS) version 26.0 and GraphPad Prism Version 8.0 software. Results Of the 764 participants included in this study, more than half had inadequate knowledge (55.0%), poor attitudes (59.4%) and bad perception about COVID-19 vaccine (55.4%). The acceptability of COVID-19 vaccine in this study was 41.9%. The acceptability of COVID-19 vaccine in Ashanti, Northern and Western North regions were 32.5%, 26.2% and 29.6% respectively. In a multivariate logistic regression analysis, receiving recent or previous vaccine such as HBV vaccine [aOR = 1.57, 95% CI (1.23–3.29), p = 0.002], having good attitude towards COVID-19 vaccine [aOR = 61.47, 95% CI (29.55–127.86), p < 0.0001] and having good perception about the COVID-19 vaccine [aOR = 3.87, 95% CI (1.40–10.72), p < 0.0001] were independently associated with higher odds of accepting COVID-19 vaccine. Conclusion More than half of residents in Ghanaian rural communities have inadequate knowledge, poor attitudes and bad perception about COVID-19 vaccine. The acceptability of COVID-19 vaccine is generally low among rural residents in Ashanti, Northern and Western North regions of Ghana. Residents living in hard-to-reach communities must be educated about the benefits of COVID-19 vaccine to achieve effective vaccination program.