Human semen quality may be declining due to environmental pollutants, occupational exposures or changes in lifestyle. Nonetheless, we still know little about the impact of those factors on male fertility. Some heavy metals, volatile organic compounds or xenoestrogens may compromise reproductive male function. This process could take place along the human life cycle and not only in certain stages of development. We review the main factors that affect human male fertility and their possible influence in current human reproduction.