Vaccine refusal or hesitancy is one of the significant issues that can have an adverse impact on people’s health and their countries’ economy. Additionally, vaccine acceptance or refusal could have a decisive role in controlling the COVID-19 pandemic. This study aims to investigate the factors affecting COVID-19 vaccine refusal and hesitancy among the general population in Saudi Arabia. The method is a cross-sectional survey using an online questionnaire, and data were collected from 1935 participants between 18 February 2021 and 1 April 2021. Out of 1935 participants aged ≥18 years and residing in Saudi Arabia, 46.9% reported their intention to receive the COVID-19 vaccine, 22.4% had received the vaccine, 9.5% did not intend to receive the vaccine, and 21.2% had not made their decision. In the multinomial logistic regression models, vaccine refusal was associated with age (OR: 0.961), nationality (OR: 0.182), monthly income of more than SAR 18,000 (OR: 2.325), chronic diseases (OR: 0.521), knowledge about the vaccine (OR: 0.937), and concerns about the vaccine (OR: 1.5). The hesitancy was associated with age (OR: 0.977), nationality (OR: 0.231), monthly income between SAR 6000 to 12,000 (OR: 0.607), chronic diseases (OR: 0.640), knowledge about the vaccine (OR: 0.907), and concerns about the vaccine (OR: 1.3). The main concerns about the vaccine were “COVID-19 vaccines are not tested enough on people”, “drug companies are interested in COVID-19 vaccine sales only”, and “COVID-19 vaccines have serious adverse effects”. Awareness programs and vaccination campaigns should consider people’s concerns and correct their misinformation.