Objective: to investigate the association between anogenital distance (AGD) and the risk of Late-onset Hypogonadism (LOH). Methods: We prospectively recruited 145 consecutive patients referred for transrectal prostate biopsy due to prostate-specific antigen (PSA) elevation or abnormal digital rectal examination, between September 2019 and December 2021. Two variants of AGD, anus to scrotum base (AGDas) and anus to dorsal insertion of penis (AGDap) were measured. Serum total testosterone levels in fasting conditions between 07.00 and 11.00 h were determined. Patients filled out the Aging Males’ Symptoms (AMS) scale questionnaire. Results: LOH was diagnosed in 29% of the participants and presented both AGDs shorter than men without LOH, with an AGDap of 116.09 mm (9.02 mm) versus 128.71 (10.76), and an AGDas 29.70 (8.01) versus 39.93 (11.64); both p < 0.05. ANCOVA test showed a significant association between the presence of LOH and both AGD measurements. Besides, AGD measurements presented adequate accuracy for the diagnosis of LOH. Conclusions: This work adds to the scientific evidence that AGD—a biomarker of fetal exposure—could provide a non-invasive measure to address LOH, providing a potential useful clinical tool.