Although several studies have explored the association of sex hormones with glucose metabolism, the association between sex hormones and body fat distribution, which is closely related to insulin resistance, has not been fully elucidated. We have tried to explore the relationship of testosterone (T) and estradiol (E2) with visceral adipose tissue (VAT) and subcutaneous adipose tissue (SAT) mass in Chinese men with different obese and metabolic statuses. Patients and Methods: A total of 128 men from the Health Management Center of the Second Xiangya Hospital, Central South University were collected and grouped in accordance with their obese and metabolic syndrome (MS) statuses: metabolically healthy nonoverweight/obese men (MHNO), metabolically healthy overweight/obese men (MHO) and metabolically unhealthy overweight/obese men (MUO). Multiple regression analyses were performed to estimate contributions of sex hormones levels to the variations of body fat distribution and the contributions of body fat distribution to the variations of sex hormone levels. Results: With fat mass parameters as independent variables, SAT had a strong negative association with T in MHNO (β = −2.772, P = 0.034), VAT was positively correlated with E2 in MHO (β = 22.269, P = 0.009), and SAT was negatively associated with T in MUO (β = −3.315, P = 0.010). With sex hormones as independent variables, E2 positively correlated with VAT (β = −176.259, P = 0.048), while T negatively correlated with VAT in MHO (β = 183.150, P = 0.029). In MUO, an inverse association of T with SAT was observed (β = −213.689, P = 0.021). Conclusion: E2 and VAT had a mutual influence, thus resulting in a vicious circle, and the negative correlation between T and VAT may be related to the decrease of the MS occurrence in the MHO group. There were bi-directional relationships between sex hormones and fat distribution in men with different obese and metabolic statuses.