This study aims to investigate the prevalence and genotype distribution of anal human papillomavirus (HPV) infection among men with different sexual orientations with or without human immunodeficiency virus (HIV) in China. A cross-sectional study was conducted during 2016–2017 in Taizhou City, Zhejiang Province. Convenient sampling was used to recruit male participants from HIV voluntary counselling and testing clinics and Center for Disease Control and Prevention. A face-to-face questionnaire interview was administered and an anal-canal swab was collected for HPV genotyping. A total of 160 HIV-positive and 113 HIV-negative men participated in the study. The prevalence of any type HPV was 30.6% for heterosexual men, 74.1% for homosexual and 63.6% for bisexual men among HIV-positive participants, while the prevalence was 8.3%, 29.2% and 23.8% respectively among HIV-negatives. The most prevalent genotypes were HPV-58 (16.9%), HPV-6 (15.6%) and HPV-11 (15.0%) among HIV-positive men, and were HPV-16 (4.4%), HPV-52 (4.4%) and HPV-6 (3.5%) among HIV-negative men. Having ever had haemorrhoids and having ever seen blood on tissue after defaecation was associated with HPV infection. One-fourth of the HPV infections in this study population can be covered by the quadrivalent vaccine in market. The highly prevalent anal HPV infection among men especially HIV-infected men calls for close observation and further investigation for anal cancer prevention.