Alcohol consumption has commonly been associated with semen parameters. However, the association between alcohol intake and semen parameters in primary and secondary infertile men remains unclear. In this study, 776 infertile men from China were grouped according to alcohol intake: abstainers, moderate drinkers (<9 units/week, up to approximately 100 g of ethanol) and heavy drinkers (≥9 units/week). Semen parameters, including semen volume, sperm concentration, total sperm count, progressive motility and normal morphology were investigated. Alcohol consumption and other lifestyle factors were assessed by questionnaire. Logistic regression models were applied. There was no significant association between alcohol consumption and semen parameters in men with primary infertility. Smaller testis volumes and lower sperm concentrations were found among moderate and heavy drinkers in the secondary infertility group than among abstainers. After adjustment for potential confounders, men with secondary infertility and heavy alcohol consumption had a higher risk of abnormal sperm concentrations (OR = 3.72; 95% CI, 1.04, 13.37). These findings suggest that alcohol intake may decrease sperm concentrations in men with secondary infertility, whereas no association was found in men with primary infertility. It may be beneficial for clinicians to advise male patients with secondary infertility who are seeking fertility treatment to avoid heavy alcohol consumption.