People with pre-exposure prophylaxis (PrEP) or living with HIV are a high-risk population for monkeypox virus (MPXV) infection. It is important to achieve high MPXV vaccination coverage rates in this group. This project used self-reporting to assess vaccine hesitancy for the smallpox vaccine and acceptance among men having sex with men with PrEP or living with HIV. In total, 52 (33.6%) participants among the 155 declared their hesitancy to be vaccinated against MPXV. Moreover, 20.7% patients with PrEP declared a hesitant attitude towards the smallpox vaccine compared to 40.2% of the HIV patients, p = 0.013. This difference remained not significant after adjustment for age (p = 0.119) and after adjustment for both age and number of different sexual partners (p = 0.406). Among PrEP people, those who expressed concerns about people getting more vaccines than needed (p = 0.012) were less likely to accept vaccination, whereas an increased number of different sexual partners during the previous month was significantly associated with acceptance of vaccination (p = 0.034). Among HIV people, those who expressed concerns about being infected by MPXV (p < 0.001), those who expressed that the smallpox vaccine should be compulsory for people at risk (p < 0.001) and those with an increased the number of different sexual partners the previous month (p = 0.018) were significantly associated with higher acceptance of MPXV vaccination. Our results suggest that vaccine strategy would be efficient in France with a communication strategy emphasizing the benefits of vaccination and the potential MPXV risk infection for health in PrEP and HIV people. Other preventive actions should be implemented, including reduction in sexual partners.