During the last decade, a number of studies have shown that, in addition to their classically described reproductive function, estrogens and androgens also regulate the immune system in teleosts. Today, several molecules are known to interfere with the sex-steroid signaling. These chemicals are often referred to as endocrine disrupting contaminants (EDCs). We review the growing evidence that these compounds interfere with the fish immune system. These studies encompass a broad range of approaches from field studies to those at the molecular level. This integrative overview improves our understanding of the various endocrine-disrupting processes triggered by these chemicals. Furthermore, the research also explains why fish that have been exposed to EDCs are more sensitive to pathogens during gametogenesis. In this review, we first discuss the primary actions of sex-steroid-like endocrine disruptors in fish and the specificity of the fish immune system in comparison to mammals. Then, we review the known interactions between the immune system and EDCs and interpret the primary effects of sex steroids (estrogens and androgens) and their related endocrine disruptors on immune modulation. The recent literature suggests that immune parameters may be used as biomarkers of contamination by EDCs. However, caution should be used in the assessment of such immunotoxicity. In particular, more attention should be paid to the specificity of these biomarkers, the external/internal factors influencing the response, and the transduction pathways induced by these molecules in fish. The use of the well-known mammalian models provides a useful guide for future research in fish.