Diets could play an important role in testicular function, but studies on how adherence to the dietary patterns influences human testicular function in Asian countries are scarce. Herein, we examined the association between testosterone-related dietary patterns and testicular function among adult men in Taiwan. This cross-sectional study recruited 3283 men who attended a private medical screening program from 2009 to 2015. Testosterone-related dietary pattern was generated by the reduced rank regression (RRR) method. The association between adherence to quartile of dietary pattern scores with sex hormones (testosterone, follicle-stimulating hormone (FSH), luteinizing hormone (LH), and estradiol (E2)) and sperm quality (sperm concentration (SC), total sperm motility (TSM), progressive motility (PRM), and normal sperm morphology (NSM)) were examined by multivariable linear regression. Hemoglobin (β = 0.57, p < 0.001), hematocrit (β = 0.17, p = 0.002), triglyceride (β = −0.84, p < 0.001), HDL-cholesterol (β = 3.58, p < 0.001), total cholesterol to HDL-cholesterol ratio (β = −0.78, p < 0.001), and uric acid (β = −10.77, p < 0.001) were highly correlated with testosterone levels. Therefore, these biomarkers were used to construct a testosterone-related dietary pattern. Highest adherence (Q4) to dietary pattern scores were negatively associated with lower testosterone in the pooled analysis (β = −0.89, p = 0.037) and normal-weight men (β = −1.48, p = 0.019). Likewise, men in the Q4 of the dietary pattern had lower SC (β = −5.55, p = 0.001) and NSM (β = −2.22, p = 0.007) regardless of their nutritional status. Our study suggesting that testosterone-related dietary pattern (rich in preserved vegetables or processed meat or fish, deep-fried foods, innards organs, rice or flour products cooked in oil, and dipping sauce, but low in milk, dairy products, legumes, or beans, and dark or leafy vegetables) was associated with a poor testicular function.