The rising incidence rate of prostate cancer (PCa) worldwide has become a public health concern. PCa has a multifactorial etiology, and the link between human papillomavirus (HPV) and PCa has been widely investigated by numerous case–control studies. This age-matched, case–control study included 143 PCa patients and 135 benign prostatic hyperplasia (BPH) patients, with prostatic specimens testing negative for malignancy, as control. Study participants were recruited from four major hospitals in Taoyuan City, Taiwan, period 2018–2020, looking into HPV infection and other PCa risk factors, including dietary habits, family history, personal lifestyle, and sexual behavior. Multiple logistic regression analysis and forward stepwise selection analysis were conducted to identify potential risk factors for PCa. HPV DNA was found in 10 of the 143 PCa cases (7%) and 2 of the 135 BPH controls (1.5%) (OR = 6.02, 95% CI = 1.03–30.3, p = 0.046). This association was slightly significant, and furthermore, high risk HPV was not found to be associated with PCa. Higher body mass index (BMI) (OR = 1.15, 95% CI = 1.05–1.27, p = 0.003), more total meat consumption (OR = 2.74, 95% CI = 1.26–5.94, p = 0.011), exhibited association to PCa. However, PCa family history only presented a statistically significant difference by forward stepwise analysis (OR = 3.91, 95% CI = 1.17–13.12, p = 0.027). While much focus has been on the association between HPV and PCa, the results of this study indicate that more efforts should be directed towards investigating dietary habits, personal lifestyle and family history as factors for PCa. These results could serve as a basis for designing PCa prevention strategies.