Objectives: The COVID-19 pandemic is among the most serious global threats, and it is still a significant concern. The people of Bangladesh are undergoing one of the world's largest vaccination drive. With the recent launch and introduction of the COVID-19 vaccines, many of us are curious about the general opinion or view of the vaccine. While the vaccine has ignited new hope in the battle against COVID-19, it has also sparked militant anti-vaccine campaigns, so the need to analyze public opinion on the COVID-19 vaccine has emerged. Methods: Traditional machine learning methods were used to obtain a benchmark result for the experiment. The recurrent neural network (RNN) algorithm was used next. Several different types of recurrent neural networks were used, including simple RNNs, Gated Recurrent Units (GRUs), and LSTMs. Finally, to achieve a more optimal result, small BERT models (Bidirectional Encoder Representations from Transformers) were used. Results: Upon study and testing on several models and methods, it can be seen that BERT model was the most accurate of the bunch, which was 84%. On the other hand, Naive Bayes was able to obtain an accuracy of 81%. Naive Bayes and BERT produced similar results in F1- Score, but the performance of Naive Bayes can improve as the dataset size grows. Conclusion: Knowing about public opinions on the COVID-19 vaccine is critical, and action must be taken to ensure that everybody understands the value of vaccination and that everybody receives the COVID-19 vaccine. Vaccination may help to develop immunity, which lowers the likelihood of contracting the disease and its consequences.