Japan has been reported as a country with high levels of vaccine hesitancy. However, a lack of comprehensive reviews studying factors for vaccine hesitancy for the COVID-19 vaccines in the Japanese context from the perspective of ethical controversy exists. Using a narrative review method, we reviewed factors associated with vaccine hesitancy to the COVID-19 vaccines and examined issues related to ethical controversy among the Japanese population. Factors associated with vaccine hesitancy include concerns about vaccine safety, suspicion of vaccine inefficacy, mistrust of the government, and low perceived threat. Factors associated with vaccine acceptance include environmental factors, factors related to Japanese cultural values, including collectivism and social norms, and positive attitudes toward information provided by authorities. Unique backgrounds in Japan are historical events such as the anti-HPV vaccine campaigns, the accessible medical system fostering high expectations of zero risk, and cultural factors of caring social norms influencing vaccine acceptance. Ethical controversies arise from preferences and practices at the individual or national level around individual rights versus public health benefits. Healthcare professionals and public health experts should continue dialoguing with the critical mass, practitioners, and policymakers, considering the ethical dilemmas surrounding individual rights and public health benefits. Insights obtained from this study indicate the need to develop tailored strategies to enhance vaccine acceptance while respecting individual autonomy within the Japanese context.