According to the World Health Organization, vaccine hesitancy was one of the ten major threats to global health in 2019, including the COVID-19 vaccine. The availability of vaccines does not always mean utilization. This is because, people have less confidence in vaccines, which resulted in vaccination hesitancy and developing global decline in vaccine intake and has caused viral disease outbreaks worldwide. Therefore, there is a need to understand people’s perceptions about the COVID-19 vaccine to help the manufacturing companies of the vaccine to improve their marketing strategy based on the rejection causes. In this paper, we used multi-class Sentiment Analysis to classify people’s opinions from extracted tweets about COVID-19 vaccines, using firstly different Machine Learning (ML) classifiers such as Logistic Regression (LR), Stochastic Gradient Descent, Support Vector Machine, K-Nearest Neighbors, Decision Tree (DT), Multinomial Naïve Bayes, Random Forest and Gradient Boosting and secondly various Deep Learning (DL) models such as Recurrent Neural Network (RNN), Long Short Term Memory (LSTM), Gated Recurrent Unit (GRU), RNN-LSTM and RNN-GRU. Then, we investigated the analysis of the negative tweets to identify the causes of rejection using the Latent Dirichlet Allocation (LDA) technique. Finally, we classified these negative tweets according to the rejection causes for all the vaccines using the same selected ML and DL models. The result of SA showed that DT gives the best performance with an accuracy of 92.26% and for DL models, GRU achieved 96.83%. Then, we identified five causes: Lack of safety, Side effect, Production problem, Fake news and Misinformation, and Cost. Furthermore, for the classification of the negative tweets according to the identified rejection causes, the LR achieved the best result with an accuracy of 89.97%. For DL models, the LSTM model showed the best result with an accuracy of 91.66%.