Vaccination coverage against Human Papillomavirus (HPV) is low compared with uptake of other vaccines in many countries, including Brazil. The aim of this study was to examine the main reasons provided by parents or guardians of a target population that did not have the first dose of HPV vaccine in a small rural Brazilian municipality, and to verify the factors associated with the reasons for non-vaccination. This is a cross-sectional study with interviews based on the Health Belief Model (HBM), conducted with parents and guardians of 177 unvaccinated children or adolescents. The outcome of interest was the main reason for not vaccinating the child/adolescent. The exposure factors of interest were knowledge about HPV and its prevention as well as sociodemographic characteristics. The main justifications for not vaccinating were lack of information (62.2%), fear or refusal (29.9%), and logistical issues (7.9%). The justifications associated with adolescents’ sex, fear, or refusal were mentioned by 39.3% (95% CI: 28.8–50.6%) of parents or guardians of girls and by 21.5% (95% CI: 13.7–31.2%) of parents or guardians of boys. The main barrier to HPV vaccination is lack of information. Further training of health professionals in clarifying the benefits of vaccination and differentiating the risks between boys and girls could encourage uptake.