Obesity and metabolic-related diseases, among which diabetes, are prominent public health challenges of the 21st century. It is now well acknowledged that pollutants are a part of the equation, especially endocrine-disrupting chemicals (EDCs) that interfere with the hormonal aspect. The aim of the review is to focus on adipose tissue, a central regulator of energy balance and metabolic homeostasis, and to highlight the significant differences in the endocrine and metabolic aspects of adipose tissue between males and females which likely underlie the differences of the response to exposure to EDCs between the sexes. Moreover, the study also presents an overview of several mechanisms of action by which pollutants could cause adipose tissue dysfunction. Indeed, a better understanding of the mechanism by which environmental chemicals target adipose tissue and cause metabolic disturbances, and how these mechanisms interact and sex specificities are essential for developing mitigating and sex-specific strategies against metabolic diseases of chemical origin. In particular, considering that a scenario without pollutant exposure is not a realistic option in our current societies, attenuating the deleterious effects of exposure to pollutants by acting on the gut-adipose tissue axis may constitute a new direction of research.