Experiencing adverse childhood experiences (ACEs) may impact personal opinions, attitudes, and judgments, which can further result in HIV-related stigma. HIV-related stigma consequentially may impact HIV preventive measures such as HIV testing, pre-exposure prophylaxis uptake, and condom use. The extent to which ACEs influence HIV-related stigma perception has not been well studied. Therefore, the study aimed to examine the association between ACEs and perceived and interpersonal HIV-related stigma among Tanzanian HIV-negative men. Quantitative survey data were obtained from the Tanzania STEP (Self-Testing Education and Promotion) project established in four wards: Mabibo, Manzese, Tandale, and Mwanyanamala. A total of 507 men responded to the ACEs and HIV-related stigma questionnaires. ACEs were operationalized as types of ACEs (environmental, physical/psychological, sexual abuse) and ACE score (0 (reference) vs. 1, 2, 3, ≥ 4). Perceived HIV-related stigma was analyzed both as a binary (HIV stigma vs. no HIV stigma) and a continuous variable. Unadjusted and adjusted multinomial logistic and linear regression models were used to assess the associations between ACEs and HIV-related stigma. ACE types were associated with HIV stigma (b = 0.237, 95% CI [0.122–0.352], p = < .0001). Findings of the adjusted multinomial logistic regression model show that experiencing one ACE (aOR = 1.9; p-value = 0.023), two ACEs (aOR = 1.8; p-value = 0.044), four or more ACEs (aOR = 4.1; p-value = < 0.0001) were associated with greater perceived HIV-related stigma. Moreover, experiencing environmental (aOR = 8.6; p-value = 0.005), physical/psychological (aOR = 2.5; p-value = 0.004), and sexual abuse (aOR = 3.4; p-value = < 0.0001) were associated with higher odds of HIV-related stigma. Our study findings suggest that those who experience childhood trauma are more likely to have a higher perception of HIV-related stigma. Intervention programs targeting HIV stigma should consider addressing ACEs entailing the behavioral and psychological impact of childhood trauma.