Introduction: Scaling up vaccination against COVID-19 is central to controlling the COVID-19 epidemic in the United States. Several vaccines are now approved for the prevention of COVID-19, but public concerns over safety and efficacy have heightened distrust and vaccine hesitancy. This is particularly concerning among people with HIV (PWH) who may be vulnerable to more severe COVID-19 disease. Here, we aimed to identify and understand COVID-19 vaccine hesitancy in a sample of PWH in the U.S. Methods: We conducted a cross-sectional online survey among PWH in the U.S. between 6 December 2020 and 8 January 2021. Measures included demographics, participants’ HIV and health-related attributes, COVID-19 history and experiences, COVID-19 vaccine-related concerns, and standardized measures of attitudes towards COVID-19 vaccines. Multivariate linear regression was used to identify factors associated with vaccine hesitancy in this sample. Results: Among the 1030 respondents, most were male (89.7%), White (66.0%), and identified as gay or lesbian (84.5%). Participants’ mean time living with HIV was 17.0 years (standard deviation (SD) = 11.1). The mean score for vaccine hesitancy was 1.5 (SD = 0.5; range: 1–5); 935 participants (90.8%) had a score greater than 1.0, indicating most participants had some degree of vaccine hesitancy. The final multivariate linear regression showed that greater vaccine hesitancy was associated with being Black (b = 0.149, p = 0.005), single (b = 0.070, p = 0.018), politically conservative (b = 0.157, p = 0.010), “anti-vaxxer” (b = 1.791, p < 0.001), concern about side effects (b = 0.226, p < 0.001), concern about safety (b = 0.260, p < 0.001), and being worried that the vaccine will not be effective (b = 0.169, p = 0.008) and they were being experimented on (b = 0.287, p < 0.001). Participants who were male White (b = −0.093, p = 0.008) and university graduates (b = −0.093, p < 0.001) and had a CD4 count of 200 cells/mm3 (b = −0.082, p = 0.048) and a liberal political orientation (b = −0.131, p < 0.001) were associated with lower vaccine hesitancy. Conclusions: Our findings provide important insights regarding COVID-19 vaccine hesitancy among PWH. Further efforts are required to understand how various social, political, and psychological factors contribute to COVID-19 vaccine hesitancy among key populations.